2
📄 来源 PDF：2.pdf
正文（按页提取）
第 1 页
Accepted for publication by IEEE Transactions on Neural Networks.
Universal Approximation to Nonlinear Operators by
Neural Networks with Arbitrary Activation Functions and
Its Application to Dynamical Systems
Tianping Chen
and Robert Chen
1
2
Abstract
The purpose of this paper is to investigate neural network capability system-
atically. The main results are: (1) Every Tauber-Wiener function is quali(cid:12)ed
as an activation function in the hidden layer of a three-layered neural network;
(2) For a continuous function to be a Tauber-Wiener function, the necessary
and su(cid:14)cient condition is that it is not a polynomial; (3) The capability of ap-
proximating nonlinear functionals de(cid:12)ned on some Banach space and nonlinear
operators has been shown, which implies that (4) we can use neural network
computation to approximate the output as a whole (not at a (cid:12)xed point) of a
dynamical system.
Key words: Approximation theory, neural networks, dynamical systems,
compact set, functional, operator.
1
2
The author is with the Department of Mathematics, Fudan University, Shanghai, P.R.China.
The author was with the Department of Electrical Engineering, University of Notre Dame, Notre
Dame, Indiana 46556, USA. He is now with VLSI Libraries, Inc., 1836 Cabrillo Ave., Santa Clara,
CA 95050.
第 2 页
1 Introduction
There have been many papers related to approximation to a continuous function
of several variables.
In 1987, Wieland and Leighton [1] dealt with the capability
of networks consisting of one or two hidden layers. Miyake and Irie [2] obtained
an integral representation formula with an integral kernel (cid:12)xed beforehand. This
representation formula is a kind of integrals, which could be realized by a three-
layered neural network. In 1989, several papers related to this topic appeared. They
all claimed that a three-layered neural network with sigmoid units on the hidden
layer can approximate continuous or other kinds of functions de(cid:12)ned on compact
set in R
. They used di(cid:11)erent methods. Carrol and Dickinson [4] used inverse
n
Radon transform. Cybenko [3] used Hahn-Banach theorem and Riesz representation
theorem. Funahashi [5] approximated Irie and Miyake's integral representation by
a (cid:12)nite sum, using a kernel which can be expressed as a di(cid:11)erence of two sigmoidal
functions. Hornik et al. [6] applied Stone-Weierstrass theorem, using trigonometric
functions.
However, in all these papers, sigmoidal functions must be assumed to be continu-
ous or monotone. Recently [9], we pointed out that the boundedness of the sigmoidal
function plays an essential role for its being an activation function in the hidden layer.
In addition to sigmoidal functions, many other functions can be used as activation
functions in the hidden layer. For example, Hornik [6] proved that any bounded non-
constant continuous function is quali(cid:12)ed to be an activation function. Mhaskar and
Micchelli [11] showed that under some restriction on the amplitude of a continuous
2
第 3 页
function near in(cid:12)nity, any non-polynomial function is quali(cid:12)ed to be an activation
function.
It is clear that all the aforementioned works are concerned with approximation to
a continuous function de(cid:12)ned on a compact set in R
(a space of (cid:12)nite dimensions).
n
However, in engineering problems such as computing the output of dynamic systems
or designing neural system identi(cid:12)ers, we often encounter the problem of approximat-
ing nonlinear functionals de(cid:12)ned on some function space, even nonlinear operators
from one function space (a space of in(cid:12)nite dimensions) to another function space
(another space of in(cid:12)nite dimensions). In [10], Sandberg gave an interesting theorem
on approximating nonlinear functionals by superposition and composition of several
linear functionals and a continuous function of one variable. Yet, two problems remain
open: 1. Can we give those linear functionals explicitly? 2. Can we approximate
nonlinear operators rather than functionals? Problem 1 is essential in application,
since otherwise we are not able to construct real networks. Problem 2 is important
in computing dynamic systems, for a dynamic system is in fact an operator. In [12],
we discussed in detail the problem of approximating nonlinear functionals de(cid:12)ned on
some compact set in C [a; b] or L
[a; b] and obtained some explicit results. However,
p
the problem of neural network's capability in approximating nonlinear operators with
its related application in computing the output as a whole of a dynamic system still
remains open. Moreover, a uni(cid:12)ed and systematic treatment of neural network ap-
proximation to continuous functions, functionals and operators is much needed but
nevertheless also remains to be an open problem.
3
第 4 页
Speci(cid:12)cally, it is quite natural to raise the following issues: (1) What is the char-
acteristic property for a continuous function in the hidden layer of a neural network?
(2) To give a neural network model to approximate nonlinear functionals de(cid:12)ned on
some compact set in C (K ), where K is some compact set in some Banach space. (3)
To give a neural network model, which can be used to approximate the output of
some dynamic system as a whole (not merely at a special point, cf. [10][12]), thus to
identify the dynamic system.
In this paper, we systematically give strong results for these issues.
The paper is organized as follows. In section 2, we review some de(cid:12)nitions and
notations.
In section 3, we show that the necessary and su(cid:14)cient condition for a
continuous function in S
(R
) (tempered distributions in R
) to be a Tauber-Wiener
0
1
1
function (for de(cid:12)nitions, see section 2) is that it is not a polynomial; and any Tauber-
Wiener function can be used as an activation function, i.e., any non-polynomial con-
tinuous function in S
(R
) is an activation function. What is more interesting is
0
1
that we show the approximation is equiuniform on any compact set in C (K ), which
is crucial in discussing approximation to continuous operators by neural networks.
In section 4, we show the capability of neural networks to approximate continuous
functionals de(cid:12)ned on some compact set in C (K ), where K is a compact set in some
Banach space; and through which we establish the capability of neural networks to
approximate continuous operators from C (K
) to C (K
). The main results in section
1
2
4 has a direct application to computing output of dynamic systems thus identifying
the systems, which is discussed in section 5.
4
第 5 页
2 Notations and De(cid:12)nitions
De(cid:12)nition 1.
A function (cid:27) : R
! R
is called a sigmoidal function, if it
1
1
satis(cid:12)es
(
lim
(cid:27) (x) = 0 ;
x!(cid:0)1
lim
(cid:27) (x) = 1 :
x!1
De(cid:12)nition 2.
If a function g : R ! R (continuous or discontinuous) satis(cid:12)es
that all the linear combinations
c
g ((cid:21)
x + (cid:18)
), (cid:21)
2 R, (cid:18)
2 R, c
2 R, i =
i=1
i
i
i
i
i
i
P
N
1; 2; : : : ; N , are dense in every C [a; b], then g is called a Tauber-Wiener function, or
simply (TW) function.
De(cid:12)nition 3.
Suppose that X is a Banach space, V (cid:18) X is called a compact
set in X , if for every sequence fx
g
with all x
2 V , there is a subsequence fx
g,
n
n
n=1
n
k
1
which converges to some element x 2 V .
It is well known that if V (cid:18) X is a compact set in X , then for any (cid:14) > 0, there is
a (cid:14) -net N ((cid:14) ) = fx
; : : : ; x
g, with all x
2 V , i = 1; : : : ; n((cid:14) ), i.e. for every x 2 X ,
1
n((cid:14))
i
there is some x
2 N ((cid:14) ) such that kx
(cid:0) xk
< (cid:14) .
i
i
X
In the sequel, we will often use the following notations.
X : some Banach space with norm k (cid:1) k
.
X
n
R
: Euclidean space of dimension n.
K : some compact set in a Banach space.
C (K ): Banach space of all continuous functions de(cid:12)ned on K , with norm kf k
=
C (K )
5
第 6 页
max
jf (x)j.
x2K
(TW): All the Tauber-Wiener functions.
n
S (R
): Schwartz functions in tempted distribution theory, i.e. rapidly decreasing
and in(cid:12)nitely di(cid:11)erentiable functions.
0
n
S
(R
): Tempered distributions, i.e.
linear continuous functionals de(cid:12)ned on
n
S (R
).
1
n
C
(R
): In(cid:12)nitely di(cid:11)erentiable functions.
1
n
n
C
(R
): In(cid:12)nitely di(cid:11)erentiable functions with compact support in R
.
c
n
C
[(cid:0)1; 1]
: All 2-periodic functions with period 2 with respect to every variable
p
x
, i = 1; : : : ; n.
i
3 Characteristics of Activation Functions
In this Section, we prove three theorems.
Theorem 1
Suppose that g is a continuous function, and g 2 S
(R
), then g 2
0
1
(T W ), if and only if g is not a polynomial.
Theorem 2
If (cid:27) is a bounded sigmoidal function, then (cid:27) 2 (T W ).
Theorem 3
Suppose that K is a compact set in R
, U is a compact set in C (K ),
n
g 2 (T W ), then for any (cid:15) > 0, there exist a positive integer N , real numbers (cid:18)
,
i
6
第 7 页
vectors !
2 R
, i = 1; : : : ; N , which are independent of f 2 C (K ) and constants
i
n
c
(f ), i = 1; : : : ; N depending on f , such that
i
N
X
jf (x) (cid:0)
c
(f )g (!
(cid:1) x + (cid:18)
)j < (cid:15)
(1)
i
i
i
i=1
holds for al l x 2 K and f 2 U . Moreover, each c
(f ) is a linear continuous functional
i
de(cid:12)ned on U .
Remark 1. Theorem 3 shows that for a function (continuous or discontinuous)
to be quali(cid:12)ed as an activation function, a su(cid:14)cient condition is that it belongs
to (T W ) class. Therefore, in order to prove that a neural network is capable of
approximating any continuous function of n variables, all we need to do is to deal
with the case n = 1, thus we have reduced the complexity of the problem in terms of its
dimensionality. Moreover, by examining the approximated function f (x
; : : : ; x
) =
1
n
f (x
; 0; : : : ; 0) = f
(x
), where f
(x
) is a continuous function of one variable, it is
1
1
1
(cid:3)
(cid:3)
straightforward to see that the condition is also a necessary one.
Remark 2.
The equiuniform convergence property in Theorem 3 will play a
crucial role in approximating nonlinear operators by neural networks.
Remark 3. When a sigmoidal function is used as an activation in a neu-
ral network, Theorem 2 shows that the only necessary condition imposed on is its
boundedness. In contrast, in almost all other papers ([1], [2], [3], [4], [5], [6], [7], [8]),
sigmoidal functions must be assumed to be either continuous or monotone.
Remark 4.
In [11], some result similar to Theorem 1 was obtained under more
restrictions imposed on g , i.e., there are positive integer N and a constant C
, such
N
7
第 8 页
that j(1 + jxj)
g (x)j (cid:20) C for all x 2 R
. This restriction is essential for [11], for
(cid:0)N
1
the proof in [11] depends heavily on a variation of Paley-Wiener Theorem. However,
in Theorem 1, we only assume that g 2 C (R
) \ S
(R
), which is weaker than the
1
0
n
assumptions used in [11].
Proof of Theorem 1. We will prove by contradiction. If all the linear combina-
P
n
tions
c
g ((cid:21)
x + (cid:18)
) are not dense in C [a; b], then Hahn-Banach extension theorem
i=1
i
i
i
and Riesz representation of linear continuous functionals show that there is a signed
Borel measure d(cid:22) with supp(d(cid:22)) (cid:18) [a; b] and
Z
1
R
g ((cid:21)x + (cid:18)) d(cid:22)(x) = 0
(2)
for all (cid:21) 6= 0 and (cid:18) 2 R
. Take any w 2 S (R
), then
1
1
Z
Z
w((cid:18)) d(cid:18)
g ((cid:21)x + (cid:18)) d(cid:22)(x) = 0:
(3)
1
1
R
R
Let (cid:21)x + (cid:18) = u and change order of integration, we have
Z
Z
u (cid:0) (cid:18)
g (u)
w((cid:18)) d(cid:22)(
) = 0
(4)
1
1
R
R
(cid:21)
which is equivalent to
^g ( ^w((cid:1))
d(cid:22)((cid:21)(cid:1))) = 0
(5)
^
where ^g represents Fourier transform of g in the sense of tempered distribution, and
(5) is also understood in the sense of distribution (see [13]). In order that the left
hand side of (5) makes sense, we have to show that ^w (t)
d(cid:22)((cid:21)t) 2 S (R
). Since
c
1
supp(d(cid:22)) (cid:18) [a; b], it is straightforward to show that
d(cid:22)(t) 2 C
(R
) and for each
c
1
1
k = 1; 2; : : :, there is a constant c
such that
k
k
@
c
j
d(cid:22)(t)j (cid:20) c
:
(6)
k
@ t
k
8
第 9 页
Consequently, ^w(t)
d(cid:22)(t) 2 S (R
).
c
1
Since d(cid:22) 6(cid:17) 0 and
d(cid:22)(t) 2 C
(R
), hence there exists some t
6= 0 with some
0
c
1
1
neighborhood (t
(cid:0)(cid:14); t
+(cid:14) ) such that
d(cid:22)(t) 6= 0 for all t 2 (t
(cid:0)(cid:14); t
+(cid:14) ). Now, if t
6= 0,
0
0
0
0
1
c
t
0
c
(cid:14)
(cid:14)
1
(cid:14)
(cid:14)
let (cid:21) =
, then
d(cid:22)((cid:21)t) 6= 0 for all t 2 (t
(cid:0)
; t
+
). Take any ^w 2 C
(t
(cid:0)
; t
+
),
t
1
1
1
0
0
c
(cid:21)
(cid:21)
2(cid:21)
2(cid:21)
then ^w(t)=
d(cid:22)((cid:21)t) 2 S (R
), and by (5)
c
1
^g ( ^w ((cid:1))) = ^g (
d(cid:22)((cid:21)(cid:1))) = 0
(7)
^w ((cid:1))
c
c
d(cid:22)((cid:21)(cid:1))
Previous argument shows that for any (cid:12)xed point t
, there is a neighborhood
(cid:3)
(cid:3)
(cid:3)
(cid:3)
(cid:3)
[t
(cid:0) (cid:17) ; t
+ (cid:17) ] such that ^g ( ^w((cid:1))) = 0 for all ^w with compact support [t
(cid:0) (cid:17) ; t
+ (cid:17) ], i.e.
supp(^g ) (cid:18) f0g. By the distribution theory, ^g is some linear combination of (cid:14) -Dirac
function and its derivatives, which is equivalent to that g is a polynomial. Theorem
1 is proved.
Proof of Theorem 2 can be found in [9]. Here we only give a brief proof for the
completeness of this paper.
Proof of Theorem 2. Without loss of generality, we can assume that [a; b] = [0; 1].
Since f is continuous on [(cid:0)1; 1] for any (cid:15) > 0, there is an integer M > 0, such that
0
00
0
00
0
00
jf (x
) (cid:0) f (x
)j < (cid:15)=4, provided that x
; x
2 [(cid:0)1; 1] and jx
(cid:0) x
j < 1=M .
Divide [(cid:0)1; 1] into 2M equal segments, each has length of 1=M . Let
(cid:0) 1 = x
< x
< : : : < x
= 0 < x
< : : : < x
= 1
(8)
0
1
M
M +1
2M
and t
=
(x
+ x
), t
= (cid:0)1 (cid:0)
. From the assumption, there exists W > 0,
i
i
i+1
(cid:0)1
2
2M
1
1
such that if u > W , then j(cid:27) (u) (cid:0) 1j <
; if u < (cid:0)W , then j(cid:27) (u)j <
. Let K > 0
2
M
2
M
1
1
9
第 10 页
be such that K (cid:1)
> W . Construct
2M
1
g (x) = f ((cid:0)M )(cid:27) (K (x (cid:0) t
)) +
[f (x
) (cid:0) f (x
)](cid:27) (K (x (cid:0) t
))
(9)
(cid:0)1
i
i(cid:0)1
i(cid:0)1
i=1
N
X
then we can prove
jg (x) (cid:0) f (x)j < (cid:15)
for all x 2 [(cid:0)1; 1]
(10)
Theorem 2 is thus proved.
Prior to proving Theorem 3, we need to establish the following lemmas.
Lemma 1
Suppose that K is a compact set in R
, f 2 C (K ), then there is a
n
continuous function E (f ) 2 C (R
), such that (1) f (x) = E (f )(x) for al l x 2 K ; (2)
n
sup
jE (f )(x)j (cid:20) sup
jf (x)j; (3) there is a constant c such that
n
x2R
x2K
sup
jE (f )(x
) (cid:0) E (f )(x
)j (cid:20) c sup
jf (x
) (cid:0) f (x
)j
(11)
0
00
0
00
0
00
0
00
jx
(cid:0)x
j<(cid:14)
x
(cid:0)x
<(cid:14)
0
00
x
;x
2K
Proof. The proof of Lemma 1 can be found in [14] (p. 175).
Lemma 2 [15]
V is a compact set in C (K ), if and only if
1. V is a closed set in C (K ).
2. There is a constant M , such that kf (x)k
(cid:20) M for al l f 2 V .
C (K )
3. V is equicontinuous, i.e.
for any (cid:15) > 0, there is a (cid:14) > 0 such that jf (x
) (cid:0)
0
00
0
00
0
00
f (x
)j < (cid:15) for al l f 2 V , provided that x
, x
2 K and kx
(cid:0) x
k
< (cid:14) .
K
10
第 11 页
Lemma 3
Suppose that K is a compact set in I
= [0; 1]
, V is a compact set in
n
n
C (K ), then V can be extended to a compact set in C
[(cid:0)1; 1]
.
p
n
Proof. By Lemmas 1 and 2, V can be extended to be a compact set V
in C [0; 1]
.
1
n
Now, for every f 2 V
, de(cid:12)ne an even extension of f as follows
1
(cid:3)
f
(x
; : : : ; x
; : : : ; x
) = f (x
; : : : ; (cid:0)x
; : : : ; x
)
(12)
1
k
n
1
k
n
then U = ff
: f 2 V
g is the required compact set in C
[(cid:0)1; 1]
.
1
p
(cid:3)
n
Lemma 4
Suppose that U is a compact set in C
[(cid:0)1; 1],
p
B
(f ; x) =
(1 (cid:0)
)
c
(f )e
(13)
R
m
X
2
jmj
(cid:11)
i(cid:25)m(cid:1)x
jmj(cid:20)R
2
R
is the Bochner-Riesz means of Fourier series of f , where m = (m
; : : : ; m
), jmj
=
1
n
2
P
n
2
i=1
i
m
jm
j
, c
(f ) are Fourier coe(cid:14)cients of f , then for any (cid:15) > 0, there is R > 0
such that
for every f 2 U and x 2 [(cid:0)1; 1]
, provided that (cid:11) > (n (cid:0) 1)=2.
n
jB
(f ; x) (cid:0) f (x)j < (cid:15)
(14)
R
Proof. The proof of Lemma 4 can be found in [13].
Proof of Theorem 3. Without loss of generality, we can assume that K (cid:18) [0; 1]
.
n
By Lemma 3, we can assume that K = [(cid:0)1; 1]
and U (cid:18) C
[(cid:0)1; 1]
. By Lemma 4,
p
n
n
for any (cid:15) > 0, there exists R > 0, such that for any x = (x
; : : : ; x
) 2 [(cid:0)1; 1]
and
1
n
n
f 2 U , there holds
X
2
jmj
(cid:15)
(cid:11)
(cid:3)
(cid:3)
j
(1 (cid:0)
)
c
(f
)exp(i(cid:25) (m
x
+ (cid:1) (cid:1) (cid:1) + m
x
)) (cid:0) f
(x
; : : : ; x
)j <
(15)
m
(cid:1)(cid:1)(cid:1)m
1
1
n
n
1
n
jmj(cid:20)R
2
R
1
n
2
11
第 12 页
By the de(cid:12)nition of the Fourier coe(cid:14)cients and evenness of f
(x), we can rewrite (15)
(cid:3)
as
X
j
d
cos((cid:25) (m
x
+ (cid:1) (cid:1) (cid:1) + m
x
) (cid:0) f
(x
; : : : ; x
)j <
(16)
m
(cid:1)(cid:1)(cid:1)m
1
1
n
n
1
n
1
n
(cid:3)
(cid:15)
jmj(cid:20)R
2
n
where d
are real numbers. It is obvious that for every x 2 [(cid:0)1; 1]
, there is a
m
(cid:1)(cid:1)(cid:1)m
1
n
p
p
unique u 2 [(cid:0)
n(cid:25)R;
n(cid:25)R], such that
u = (cid:25)m (cid:1) x = (cid:25) (m
x
+ : : : + m
x
) :
(17)
1
1
n
n
where m = (m
; : : : ; m
). Since cos(u) is a continuous function in [(cid:0)
n(cid:25)R;
n(cid:25)R]
1
n
p
p
and g 2 (T W ), we can (cid:12)nd an integer M , real numbers s
, (cid:17)
and (cid:24)
, j = 1; : : : ; M ,
j
j
j
such that
M
X
(cid:15)
j
s
g ((cid:24)
u + (cid:17)
) (cid:0) cos(u)j <
(18)
j
j
j
j=1
2L
p
p
P
holds uniformly for all u 2 [(cid:0)
n(cid:25)R;
n(cid:25)R], where L =
jd
j. Thus
jmj(cid:20)R
m
;:::;m
1
n
M
X
(cid:15)
j
s
g ((cid:24)
(cid:25) (m (cid:1) x) + (cid:17)
) (cid:0) cos((cid:25)m (cid:1) x)j <
(19)
j
j
j
2L
j=1
n
holds for x 2 [(cid:0)1; 1]
. Substituting (19) into (16), we conclude that there exist N ,
c
, (cid:18)
2 R, !
2 R
, i = 1; : : : ; N , such that
i
i
i
n
N
X
(cid:3)
(cid:3)
(cid:3)
jf
(x) (cid:0)
c
(f
)f
(!
(cid:1) x + (cid:18)
)j < (cid:15)
(20)
i
i
i
i=1
is true for all x 2 [(cid:0)1; 1]
and g 2 U . Thus
n
N
X
jf (x) (cid:0)
c
(f )g (!
(cid:1) x + (cid:18)
)j < (cid:15)
i
i
i
i=1
is true for all x 2 [0; 1]
and f 2 V .
n
12
第 13 页
It is obvious that for each (cid:12)xed m = (m
; : : : ; m
), the Fourier coe(cid:14)cient
1
n
Z
Z
1
1
(cid:1) (cid:1) (cid:1)
e
g (x
; : : : ; x
) dx
(cid:1) (cid:1) (cid:1) dx
1
n
1
n
(cid:0)i(cid:25)(m
x
+(cid:1)(cid:1)(cid:1)+m
x
)
1
1
n
n
(cid:0)1
(cid:0)1
is a continuous functional de(cid:12)ned on U (and also a continuous functional de(cid:12)ned on
V ), and c
(f ), being a (cid:12)nite linear combination of the Fourier coe(cid:14)cients of f
, is
i
(cid:3)
surely a continuous functional de(cid:12)ned on V . The proof of Theorem 3 is completed.
4 Approximation to Nonlinear Continuous
Functionals and Maps
In this section, we will discuss the problem of approximating nonlinear continuous
functionals and operators by neural network computation. The main results are as
follows.
Theorem 4
Suppose that g 2 (T W ), X is a Banach Space, K (cid:18) X is a compact
set, V is a compact set in C (K ), f is a continuous functional de(cid:12)ned on V , then
for any (cid:15) > 0, there are an positive integer N , m points x
; : : : ; x
2 K , and real
1
m
constants c
, (cid:18)
, (cid:24)
, i = 1; : : : ; N , j = 1; : : : ; m, such that
i
i
ij
N
m
X
X
jf (u) (cid:0)
c
g (
(cid:24)
u(x
) + (cid:18)
)j < (cid:15)
(21)
i
ij
j
i
i=1
j=1
holds for al l u 2 V .
Theorem 5
Suppose that g 2 (T W ), X is a Banach Space, K
(cid:18) X , K
(cid:18) R
1
2
n
are two compact sets in X and R
respectively, V is a compact set in C (K
), G is a
1
n
13
第 14 页
nonlinear continuous operator, which maps V into C (K
), then for any (cid:15) > 0, there
2
are positive integers M , N , m, constants c
, (cid:16)
, (cid:24)
2 R, points !
2 R
, x
2 K
,
i
ij
k
k
j
1
k
k
n
i = 1; : : : ; M , k = 1; : : : ; N , j = 1; : : : ; m, such that
N
M
m
X
X
X
k
k
k
jG(u)(y ) (cid:0)
c
g (
(cid:24)
u(x
) + (cid:18)
)g (!
(cid:1) y + (cid:16)
)j < (cid:15)
(22)
i
ij
i
j
k
k
k=1
i=1
j=1
holds for al l u 2 V and y 2 K
.
2
Following lemmas are well known and will be used in the proof of Theorems 4 and
5.
Lemma 5
Let X be a Banach Space and K (cid:18) X , then K is a compact set if and
only if the fol lowing two conditions are satis(cid:12)ed simultaneously: (1) K is a closed set
in X ; (2) for any (cid:14) > 0, there is a (cid:14) -net N ((cid:14) ) = fx
; : : : ; x
g, i.e. for any x 2 K ,
1
n((cid:14))
there constitute an x
2 N ((cid:14) ) such that kx (cid:0) x
k
< (cid:14) .
k
k
X
Lemma 6
If V (cid:18) C (K ) is a compact set in C (K ), then it is uniformly bounded
and equicontinuous, i.e. (1) There is A > 0 such that ku(x)k
(cid:20) A for al l u 2 V
C (K )
and (2) for any (cid:15) > 0, there is (cid:14) > 0 such that ju(x
) (cid:0) u(x
)j < (cid:15) for al l u 2 V ,
0
00
provided that kx
(cid:0) x
k
< (cid:14) .
X
0
00
Now pick a sequence (cid:15)
> (cid:15)
> (cid:1) (cid:1) (cid:1) > (cid:15)
! 0, then we can (cid:12)nd another sequence
1
2
n
(cid:14)
> (cid:14)
> (cid:1) (cid:1) (cid:1) > (cid:14)
! 0, such that jf (u) (cid:0) f (v )j < (cid:15)
for all f 2 V , provided that
1
2
n
k
u; v 2 V and ku (cid:0) vk
< 2(cid:14)
, for f is a continuous functional de(cid:12)ned on a compact
C (K )
k
set V .
14
第 15 页
By Lemma 6, we can also (cid:12)nd (cid:17)
> (cid:17)
> (cid:1) (cid:1) (cid:1) (cid:17)
! 0 such that ju(x
) (cid:0) u(x
)j < (cid:14)
1
2
n
k
0
00
for all u 2 V , whenever x
; x
2 K and kx
(cid:0) x
k
< (cid:17)
.
K
k
0
00
0
00
By induction and rearrangement, we can (cid:12)nd a sequence fx
g
with each x
2 K
i
i
i=1
1
and a sequence of positive integers n((cid:17)
) < n((cid:17)
) < : : : < n((cid:17)
) ! 1, such that the
1
2
k
(cid:12)rst n((cid:17)
) elements N ((cid:17)
) = fx
; : : : ; x
g is an (cid:17)
-net in K .
k
k
1
k
n((cid:17)
)
k
For each (cid:17)
-net, de(cid:12)ne functions
k
(
kx(cid:0)x
k
j
k
(cid:3)
j
X
k
1 (cid:0)
if kx (cid:0) x
k
(cid:20) (cid:17)
(cid:17)
k
T
(x) =
(23)
(cid:17)
;j
k
0
otherwise
and
(cid:3)
T
(x)
(cid:17)
;j
k
T
(x) =
(24)
(cid:17)
;j
k
P
n((cid:17)
)
k
(cid:3)
j=1
(cid:17)
;j
k
T
(x)
for j = 1; : : : ; n((cid:17)
). It is easy to verify that fT
(x)g is a partition of unity, i.e.
k
(cid:17)
;j
k
0 (cid:20) T
(x) (cid:20) 1
(25)
(cid:17)
;j
k
n((cid:17)
)
k
X
T
(x) (cid:17) 1
(26)
(cid:17)
;j
k
j=1
T
(x) = 0
if kx (cid:0) x
k
> (cid:17)
:
(27)
(cid:17)
;j
k
j
X
k
For each u 2 V , de(cid:12)ne a function
n((cid:17)
)
k
X
u
(x) =
u(x
)T
(x)
(28)
(cid:17)
k
j
(cid:17)
;j
k
j=1
(cid:3)
1
and sets V
= fu
: u 2 V g and V
= V [ ([
V
). We then have the following
(cid:17)
(cid:17)
(cid:17)
k
k
k=1
k
result.
Lemma 7
15
第 16 页
1. For each (cid:12)xed k , V
is a compact set in a subspace of dimension n((cid:17)
) in C (K ).
(cid:17)
k
k
2. For every u 2 V , there holds
(cid:3)
3. V
is a compact set in C (K ).
ku (cid:0) u
k
< (cid:14)
(29)
(cid:17)
C (K )
k
k
Proof. We will prove the three propositions individually as follows.
1. For a (cid:12)xed k , let u
, i = 1; 2; : : : ; be a sequence in V
and u
be a sequence
(cid:17)
k
(cid:17)
k
(i)
(i)
in V , such that
n((cid:17)
)
k
X
(i)
(i)
u
=
u
(x
)T
(x) :
(30)
(cid:17)
k
j
(cid:17)
;j
k
j=1
Since V is a compact set, there is a subsequence u
(x), which converges to
(i
)
l
some u 2 V , then it is obvious that u
(x) converges to u
(x) 2 V
, i.e. V
(cid:17)
k
(cid:17)
(cid:17)
(cid:17)
k
k
k
(i
)
l
is a compact subset in C (K ).
2. By the de(cid:12)nition and the property of unity partition, we have
u(x) (cid:0) u
(x) =
[u(x) (cid:0) u(x
)]T
(x)
(cid:17)
k
j
(cid:17)
;j
k
n((cid:17)
)
k
X
j=1
X
=
[u(x) (cid:0) u(x
)]T
(x) :
(31)
j
(cid:17)
;j
k
Consequently,
kx(cid:0)x
k
(cid:20)(cid:17)
j
X
k
n((cid:17)
)
k
X
ku(x) (cid:0) u
(x)k
(cid:20) (cid:14)
T
(x) = (cid:14)
for all u 2 V :
(32)
(cid:17)
X
k
(cid:17)
;j
k
k
k
j=1
3. Suppose fu
g
is a sequence in V
.
If there is a subsequence fu
g
of
j=1
l=1
i
1
(cid:3)
i
1
l
i
1
i
l
fu
g
with all u
2 V , l = 1; : : : ; then by the fact that V is compact, there is
j=1
16
第 17 页
a subsequence of fu
g
, which converges to some u 2 V . Otherwise, to each
l=1
i
1
l
i
i
u
, there corresponds a positive integer k (i) such that u
= v
. There are two
(cid:17)
k(i)
possibilities: (i) We can (cid:12)nd in(cid:12)nite i
and a (cid:12)xed k
such that (cid:17)
= (cid:17)
=
l
0
k(i
)
k(i
)
1
2
(cid:1) (cid:1) (cid:1) = (cid:17)
= (cid:1) (cid:1) (cid:1) = (cid:17)
, i.e. u
2 V
for all i
. By proposition 1. of this lemma,
k(i
)
l
k
0
(cid:17)
k
0
l
i
l
V
is a compact set, there is a subsequence of fv
g, which converges to
i
n((cid:17)
)
k
0
n((cid:17)
)
k(i)
i
some v 2 V
, i.e. there is a subsequence of fu
g converging to v 2 V
.
n((cid:17)
)
k
0
n((cid:17)
)
k
0
(ii) There are sequences i
< i
< : : : ! 1 and k (i
) < k (i
) < : : : ! 1 such
1
2
1
2
that u
2 V
. Let v
2 V be such that
i
l
i
l
n((cid:17)
)
k(i
)
l
n((cid:17)
)
k(i
)
l
X
i
l
i
l
u
(x) =
v
(x
)T
(x) :
(33)
j
(cid:17)
k(i
);j
l
j=1
Since V
2 V and V is compact, we see that there is a subsequence of fv
g
,
i
l
i
1
l
l=1
which converges to some v 2 V . By the proposition 2. of this lemma, the cor-
responding subsequence of fu
g
also converges to v . Thus the compactness
l=1
i
1
l
(cid:3)
of V
is proved.
Proof of Theorem 4. By Tietze Extension Theorem, we can de(cid:12)ne a continuous
functional on V
such that
(cid:3)
(cid:3)
f
(x) = f (x)
if x 2 V
(34)
Because f
is a continuous functional de(cid:12)ned on the compact set V
, therefore for
(cid:3)
(cid:3)
any (cid:15) > 0, we can (cid:12)nd a (cid:14) > 0 such that jf
(u) (cid:0) f
(v )j < (cid:15)=2 provided that u; v 2 V
(cid:3)
(cid:3)
(cid:3)
and ku (cid:0) vk
< (cid:14) .
C (K )
Let k be (cid:12)xed such that (cid:14)
< (cid:14) , then by (29) for every u 2 V ,
k
ku (cid:0) u
k
< (cid:14)
(35)
(cid:17)
X
k
k
17
第 18 页
which implies
for all u 2 V .
(cid:3)
(cid:3)
jf
(u) (cid:0) f
(u
)j < (cid:15)=2
(36)
(cid:17)
k
By proposition 1. of Lemma 7, we see that f
(u
) is a continuous functional
(cid:17)
k
(cid:3)
de(cid:12)ned on the compact set V
in R
. By Theorem 3, we can (cid:12)nd N , c
, (cid:24)
, (cid:18)
,
(cid:17)
k
i
ij
i
n((cid:17)
)
k
i = 1; : : : ; N , j = 1; : : : ; n((cid:17)
), such that
k
(cid:3)
N
k
n((cid:17)
)
X
X
jf
(u
) (cid:0)
c
g (
(cid:24)
u(x
) + (cid:18)
)j < (cid:15)=2 :
(37)
(cid:17)
i
ij
j
i
k
i=1
j=1
Combining it with (36), we conclude that
N
m
X
X
jf (u) (cid:0)
c
g (
(cid:24)
u(x
) + (cid:18)
)j < (cid:15)
(38)
i
ij
j
i
i=1
j=1
where m = n((cid:17)
). Thus, Theorem 4 is proved.
k
Proof of Theorem 5.
From the assumption that G is a continuous operator
which maps a compact set V in C (K
) into C (K
), it is straightforward to prove that
1
2
the range G(V ) = fG(u) : u 2 V g is also a compact set in C (K
). By Theorem 3,
2
for any (cid:15) > 0, there are a positive integer N , real numbers c
(G(u)) and (cid:16)
, vectors
k
k
!
2 R
, k = 1; : : : ; N , such that
k
n
N
X
jG(u)(y ) (cid:0)
c
(G(u))g (!
(cid:1) y + (cid:16)
)j < (cid:15)=2
(39)
k
k
k
k=1
holds for all y 2 K
and u 2 V .
2
Since G is a continuous operator, combining with the last proposition of Theorem
3, we conclude that for each k = 1; : : : ; N , c
(G(u)) is a continuous functional de(cid:12)ned
k
18
第 19 页
on V . Repeatedly applying Theorem 4, for each k = 1; : : : ; N , we can (cid:12)nd positive
integers N
, m
, constants c
, (cid:24)
, (cid:18)
2 R and x
2 K
, i = 1; : : : ; N
, j = 1; : : : ; m
,
k
k
j
1
k
k
i
ij
i
k
k
k
such that
N
m
k
k
X
X
k
k
k
(cid:15)
jc
(G(u)) (cid:0)
c
g (
(cid:24)
u(x
) + (cid:18)
)j <
(40)
k
j
i
ij
i
i=1
j=1
2L
holds for all k = 1; : : : ; N and u 2 V , where
N
X
L =
sup
jg (!
(cid:1) y + (cid:16)
)j :
(41)
k
k
k=1
y2K
2
Substituting (40) into (39), we obtain that
N
N
m
k
k
X
X
X
k
k
k
jG(u)(y ) (cid:0)
c
g (
(cid:24)
u(x
) + (cid:18)
)g (!
(cid:1) y + (cid:16)
)j < (cid:15)
(42)
i
ij
i
j
k
k
k=1
i=1
j=1
holds for all u 2 V and y 2 K
.
2
Let M = max
fN
g, m = max
fm
g and for all N
< i (cid:20) M , let c
= 0. For all
k
k
k
k
k
i
k
m
< j (cid:20) m, let (cid:24)
= 0. Thus (42) can be rewritten as
k
ij
k
N
M
m
X
X
X
k
k
k
jG(u)(y ) (cid:0)
c
g (
(cid:24)
u(x
) + (cid:18)
)g (!
(cid:1) y + (cid:16)
)j < (cid:15)
(43)
i
ij
i
j
k
k
k=1
i=1
j=1
holds for all u 2 V and y 2 K
. This completes the proof of Theorem 5.
2
A graphical representation of Theorem 5 is shown in Fig. 1.
5 Application to Nonlinear Dynamical Systems
In [12], we discussed the problem of approximating the output of a dynamical
system at a (cid:12)xed point (or time) by neural networks. As a direct application of
Theorem 5, we can use neural networks to approximate the output as a whole of a
19
第 20 页
nonlinear dynamical system. Indeed, built upon the several keystone theorems proved
earlier in Section 4, our result on this topic follows naturally.
The signi(cid:12)cance of the previous results lies in that we can use neural networks to
identify a system (linear or nonlinear). The procedure is as follows:
Let a system be V = K U , where U is the input, V is the output and K is the
system to be identi(cid:12)ed.
Suppose that according to some prior knowledge or experiments, we know sev-
eral input-output relationships V
= K U
; : : : ; V
= K U
. Generally, they can be
1
1
n
n
expressed by discrete data sets fu
(x
); s = 1; : : : ; n; j = 1; : : : ; mg, fv
(y
); s =
s
j
s
l
1; : : : ; n; j = 1; : : : ; Lg. Using these data, and by Theorem 5, we can construct a
functional
L
n
N
M
m
X
X
X
X
X
E =
jV
(y
) (cid:0)
C
g (
(cid:24)
u
(x
) + (cid:18)
)g (!
(cid:1) y
+ (cid:16)
)j
(44)
s
l
s
j
k
l
k
i
i;j
i
k
k
k
2
l=1
s=1
k=1
i=1
j=1
Parameters C
, (cid:24)
, (cid:18)
, !
, (cid:16) can be determined by minimizing E (for example, by
i
i;j
i
k
k
k
k
using back-propagation algorithm). Then the equation
N
M
m
X
X
X
k
k
k
v (y ) =
C
g (
(cid:24)
u(x
) + (cid:18)
)g (!
(cid:1) y + (cid:16)
)
(45)
i
i;j
i
j
k
k
k=1
i=1
j=1
can be viewed as an approximant of V (y ) = (K U )(y ), and so identi(cid:12)es the system
K .
If the system is linear, then E , V (y ) can be simpli(cid:12)ed as
L
n
N
M
m
X
X
X
X
X
k
2
E =
jV
(y
) (cid:0)
(cid:24)
u(x
)g (!
(cid:1) y
+ (cid:16)
)j
(46)
s
l
j
k
l
k
i;j
l=1
s=1
k=1
i=1
j=1
20
第 21 页
N
M
m
X
X
X
k
v (y ) =
(cid:24)
u(x
)g (!
(cid:1) y + (cid:16)
):
(47)
i;j
j
k
k
k=1
i=1
j=1
The larger the values of n, L, m are, the better accuracy we will obtain for this
approximation.
Therefore, we have pointed to a way of constructing neural network models for
identifying dynamic systems.
Acknowledgements. The authors wish to express their gratefulness to the re-
viewers for their valuable comments and suggestions on revising this paper.
6 Conclusion
In this paper, the problem of approximating functions of several variables, functionals
and nonlinear operators are thoroughly studied. The necessary and su(cid:14)cient condi-
tion for a continuous function in S
(R
) to be quali(cid:12)ed for an activation function is
0
1
given, which is a broad generalization of previous results ([1]- [8], especially [11]). It
is also pointed out that to prove neural network approximation capability, one needs
only to treat the one dimensional case. As applications, we show how to construct
neural networks to approximate the output of a dynamical system as a whole, not
merely at a (cid:12)xed point, thus show the capability of neural network in identifying dy-
namic systems. Moreover, we point out that using existing algorithms in literatures
(for example, back-propagation algorithm), we can determine those parameters in the
network, i.e. identify the system.
21
第 22 页
References
[1] A. Wieland and R. Leighten, \Geometric Analysis of Neural Network Capacity,"
in IEEE First ICNN. 1, pp. 385-392 (1987).
[2] B. Irie and S. Miyake, \Capacity of Three-layered Perceptrons," in IEEE ICNN
1, pp. 641-648, (1988).
[3] G. Cybenko, \Approximation by Superpositions of a Sigmoidal Function," in
Math. of Control, Signals and Systems, Vol. 2, No. 4, pp. 303-314 (1989).
[4] S. M. Carroll and B. W. Dickinson, \Construction of Neural Nets using Radon
Transform," in IJCNN Proc. I, pp. 607-611 (1989).
[5] K. Funahashi, \On the Approximate Realization of Continuous Mappings by
Neural Networks," Neural Networks, pp. 183-192, Vol. 2, (1989).
[6] K. Hornik, M. Stinchcombe and H. White, \Multi-layer Feedforward Networks
are Universal Approximators," Neural Networks, Vol. 2, pp. 359-366 (1989).
[7] K. Hornik, \Approximation Capabilities of Multilayer Feedforward Networks,"
Neural Networks, Vol. 4, pp. 251-257 (1991).
[8] V. Y. Kreinovich, \Arbitrary Nonlinearity is Su(cid:14)cient to Represent All Functions
by Neural Networks: a Theorem," Neural Networks, Vol. 4, pp. 381-383 (1991).
[9] Tianping Chen, Hong Chen and Ruey-wen Liu, \A Constructive Proof of Cy-
benko's Approximation Theorem and Its Extensions," pp. 163 - 168 in Computing
Science and Statistics (editors LePage and Page), Proc. of the 22nd Symposium
on the Interface (East Lansing, Michigan, May 1990), Springer-Verlag, ISBN
0-387-97719-8. Also submitted for publication.
[10] I. W. Sandberg, \Approximations for Nonlinear Functionals," IEEE Trans. on
Circuits and Systems, Vol. 39, No. 1, pp. 65-67, Jan. (1992).
22
第 23 页
[11] H.N. Mhaskar and C.A. Micchelli, \Approximation by Superposition of Sigmoidal
and Radial Basis Functions," Advances in Applied Mathematics, Vol. 13, pp. 350-
373, (1992).
[12] Tianping Chen and Hong Chen, \Approximation to Continuous Functionals by
Neural Networks with Application to Dynamical Systems," accepted by IEEE
Trans. on Neural Networks, to appear.
[13] E. M. Stein and G. Weiss, Introduction to Fourier Analysis on Euclidean Spaces,
Princeton University Press, (1971).
[14] E. M. Stein, Singular Integrals and Di(cid:11)erentiability Properties of Functions,
Princeton University Press, (1970).
[15] J. Diedonne, Foundation of Modern Analysis, Academic Press : New York and
London (1969), p. 142.
23
第 24 页
INPUT 2: y = [ y y . . . y ]
1
2
n
y
1
y2
.
.
.
.
.
.
.
11
12
1n
y
n
n
2
1
2
.
.
.
.
.
.
.
g
x
g
x
. . . . . . .
N
n
N
g
x
.
.
.
OUTPUT :
Approx. to G(u)(y)
1
c1
g
1
cM
.
.
.
.
.
.
.
g
2
c1
g
2
cM
g
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
N
c1
g
.
.
.
.
.
.
.
N
cM
g
1
1
.
.
.
.
.
.
.
1
1m
1
12
x 1
11
1
M1
1
M
2
1
.
.
.
.
.
.
.
2
M
N
1
.
.
.
.
.
.
.
N
M
N
M2
N
1m
N
Mm
u(x )
1
u(x )
2
.
.
.
.
.
.
.
u(x )
m
Sampling Device
INPUT 1: u(x)
Figure 1: A neural network architecture of approximation to nonlinear operator
G(u)(y )
24
q
S
S
S
S
S
q
S
S
q
S
q
S
S
q
S
q
S
S
x
x
x
x
x
x
w
w
w
w
w
z
z
z