1
📄 来源 PDF：1.pdf
正文（按页提取）
第 1 页
A Constructive Proof and An Extension of Cybenko's Approximation Theorem
Tianping Chen
Hong Chen and Ruey-wen Liu
Dept. of Mathematics
Fudan University
Shanghai, China
Dept. of Electrical Engineering
University of Notre Dame
Notre Dame, IN 46556, USA
Abstract.
In this paper. we present a constructive proof of
approximation by superposition of sigmoidal functions. We
point out a sufficient condition that the set of finite linear
combinations of the form ,L<l.cr(yj .X+9.) is dense in C(t). is
J
the boundedness of the sigmoidal function cr(x). Moreover.
we show that if the set of fmite linear combinations of the
J
form :Ec.O)(~.x+TJ.). where 0) is a univariate function. is dense
J
J
J
in LP[a,b] (l:::;p<oo) (or qa,b]) for any finite a.b. then the set
of finite linear combinations of the form ,Lc.0)(yj .x+9.) is
dense in LP(t) (or C(t». An extension in another direction
is also presented in Theorem 4 of this paper.
J
J
Key words. Constructive approximation. Neural networks,
Sigmoidal functions.
1. Introduction
Recently, Cybenko[1] stated that if O'(x) is a scalar
continuous sigmoidal function, then the set of linear
combinations of the form
well-known Kolmogorov's resolution to Hilbert's 13th
Problem. In his work[2], Kolmogorov showed that any
continuous function of n variables has an ex act
representation in terms of finite superpositions and
compositions of a finite number of univariate functions. In
[1], it is shown that any continuous functions on the
compact set [0,1] can be approximated in terms of finite
superpositions of a single sigmoidal univariate function as
closely as desired.
Due to its theoretical value and its variety of
applications in neural networks and other fields, it is
important to give a constructive proof of this problem,
which is not available at the present time. In this paper, we
will provide such a constructive proof. Furthermore, we
will show that the continuity assumption imposed on the
sigmoidal functions, which was, in fact, strictly required in
[1], is unnecessary.
Instead, the boundedness of the
sigmoidal function plays an essential role. In some sense,
the boundedness of the sigmoidal function is necessary and
sufficient for the validity of the approximation theorem. We
will also give various generalizations and extensions of our
result.
Definition.
function,
if
the
0': lR~lR is called a sigmoidal
limits lim O'(x), lim O'(x) exist, and
X --+-00
x ~oo
limO'(x)=I, lim O'(x)=O.
x-r-oo
X~
(1)
where x, YjE JR.n, x'Yj is their inner product, and Clj ' 9jE JR.,
is dense in the space of all continuous functions on the
hypercube ][n, denoted by C(][n). Similar problems were
also considered in [4],[5],[6] and [7].
This result, not only settles an long-standing
problem on the realizability of IE C(][n) by single hidden
layer feedforward artificial neural networks, but
mathematically is a suitable practical substitution of the
C. Page et al. (eds.), Computing Science and Statistics
© Springer-Verlag New York, Inc. 1992
Note: 0' need not be continuous.
1
Example. O'(x) = - - -
l+exp(-x)
is a sigmoidal function, also
1
O'(x) = { 0
x~O
x<O
is a sigmoidal function.第 2 页
164
T. Chen, H. Chen and R. Liu
2. Main Results
We claim: If(x)-g(x)ke holds for all xe (-00,00).
Theorem 1. If O'(x) is a bounded sigmoidal
function, and f(x) is a continuous function on (-00,00), for
which lim f(x)=A and limf(x)=B, where A, B are constants,
x-+-
x~
then for any £>0, there exist N, Ci. Yb 9i• such that
N
I f(x) - LCiO'(YiX+9U I <e
i=1
(2)
holds for all xe(-oo,oo).
(i) If x<-M, then If(x)-f(-M)k~,
'Dl
Ig(x}-f(-M)I ~ L1f(xi}-f(Xi-I)110'(K(X-ti-I»1
i=l
'Dl
< L ~ ~2 = ~
i=l
In other words, the set of finite linear combinations
Consequently If(x}-g(x)ke.
of the form L CiO'(YiX+9i) is dense in the space of
N
i-I
continuous functions satisfying the condition that the limits
lim f(x) and limf(x) exist.
x ..... -
x-+-oo
Proof
The proof is constructive. From tile
assumption, for any £>0, we can find M>O, such that If(x}-
AI<~ if x<-M; If(x)-BI~ if x>M; and If(x'}-f(x")I~
if Ix'I~M, Ix"I~M, and Ix'-x"I~~ .
Divide [-M,M] into 2~ equal segments, each has
1
length of M' and let
Let t.=*<21 X.+X. 1) be the center of the interval
1
1
1+
[X.,X. 1]'
1 1+
Now, construct
N
g(x) = f(-M) + L[f(xi}-f(Xi_l)]O'(K(x-~_I»
(4)
i=l
where N = 2M2.
From the assumption, there exists W>O, such that
ifu>W then IO'(U)-ll~, ifu< -W then IO'(U)I~. Let
1(>0 such that K·2M>W.
1
(ii) If x>M, then If(x)-f(M)k~,
'Dl
Ig(x)-f(M)1 ~ L1f(xu-f(Xi-I)110'(K(x-ti-I»-11
i=l
'Dl
< L ~ ~2 = ~
i=l
Consequently If(x)-g(x)ke.
(iii) Consider xe [Xk-I,Xk], then IX-~_II ~ 2~ if
i=k; Ix-ti_II> 2~ if i*k.
Furthermore,
if i<k, then
K(x-ti_I»W, and hence 100(K(x-ti_I»-1I < ~2; if i>k,
then K(x-ti_O<-W, and hence IO'(K(X-ti-I»I < M2 .
1
Consequently, we have
I g(x) -f(-M) - [f(xk>-f(Xk-I)] O'(K(x-~_I»-
k-l
L[f(Xi}-f(Xi-I)] I
i=l
k-l
~ L1f(xi}-f(Xi-I)110'(K(X-tk_I»-11
i=l
~
+ Llf(xJ-f(Xi-I)IIO'(K(x-~_I» I
i=lt+l
(5)
(6)
第 3 页
k-1
~
<~~_l + ~~_l
£..J4 M2
-£..J4 M2
i=k+1
i=1
£
<-
2
It is clear that
f(-M)+[f(x0-f(Xk-l)] O'(K(x-tk_l»
k-1
+ ~)f(xj}-f(Xi-l)]
i=1
which, combined with the previous inequality, yields
Ig(x)-f(x)1 < 2" + If(x)-f(Xk-l)I
£
+ If(x0-f(Xk-l)1I0'(K(x-tk_l»1
Cybenko '8 Approximation Theorem
165
I f(x) - LCiO'(YiX+9i) I <E
N
i=1
(8)
holds for all XE [a,b]. In other words, the set of finite linear
N
combinations of the form LCiO'(YiX+9J is dense in C[a,b].
Furthermore, in this case, we can choose all Yi>O. This is
because the constant function on [a,b] can be approximated
i=1
N
by ~ CiO'(YiX+9J with all YiPositive.
f:r'
Theorem 2_ If f(x) is a continuous function on
lin=[O,l]n, O'(u) is a bounded sigmoidal function. Then for
any E>O, there exist N,Ci,9iE JR, YiE JRn, such that for any
xElin
N
I f(x) - LCiO'(Yi-X+9i) I <E
i=1
(9)
where x-Y is the inner product of x and y.
In other words,
the set of finite linear combinations of the form
N
Here, we have assumed, without loss of generality, 100(x)I~1
for XE (-00,00).
LCiO'(Yi-X+9i) is dense in C[O,I]n.
i=1
To complete the proof of the theorem, we need
only to take care of f(-M), or to show in addition that the
constant 1 can be approximated by the linear combinations
LCiO'(YiX+9J.
Proof.
Extend f(x) to be a function g(x) on
.lfn=[-I,I]n according to the following rule: g(x)=f(x) if
XE lin, and g(Xl , ... ,-Xk, ... ,Xn)=g(x 1 , ... ,Xk, ... ,xn). Thus
g(x) can be thought of as a 2-periodic even function with
respect to every variable Xi, i=I, ... ,n.
Let N,M,xi,ti,K be the same constants as before,
then it is easy to verify that
N
~ L[O'(K(x-ti-l»+O'(-K(x-ti-l))]
i=1
(7)
is the required sum. The proof is just a repetition of the
previous procedure, and the details are omitted.
Let cm 1' ... ,m n be the Fourier coefficients of
g(Xl, ... ,Xn) on .lfn. By a well known result on Bochner
Riesz Means ([3]' page 256): for any E>O, there exists R,
such that for any X=(Xl, ... ,Xn)E.lfn,
Q.E.D.
(10)
Corollary_ If f(x) is a continuous function on
some finite interval [a,b], and O'(x) is a bounded sigmoidal
function, then for any E>O, there exist N, Ci, Yi, 9;, such
that
By the definition of the Fourier coefficients and the
evenness of g(x), we can rewrite the previous inequality as
第 4 页
166 T. Chen, H. Chen and R. Liu
I I, dm1 m cos(mlxl+ ... +mnxrJ - g(xl, ... ,xn)1 < -2
•...• n
ImlSR
£
validity of the the Theorems. The boundedness of cr(x) is
sufficient for Theorems 1 and 2 to be true.
where dm1 •...• IDn are real numbers.
It can be shown that if cr(x) is a
measurable but unbounded sigmoidal function, then
Remark 2.
(11)
LCicr(Yi·X+9i) may not necessarily be dense in C(In). In
fact, we can derme
It is obvious that for any xe In, there is a unique
n
n
ue [-I, Imil, I, Imil] such that
i=1
i=1
u=m·x=mlxl+···+mnxn
Because cos(u) is a continuous function on
n
n
[-I, Imil, I, Imil], by the Corollary of Theorem 1, we can
i=1
i=1
.
fmd N ,S '~j' 11 j , such that
m m J:m m
if'
IlSjcr(~jU+11j)-COS(U) I
J=1
£
< 2L
n
n
holds uniformly for ue [-I, I m ii, I, Imil], where
m=(mlo···,mn> and L= I,ldm1 m I. Thus
ImlSR
•...• n
i=1
i=1
if'
IlSjcr(~jm ·x+11j)-cos(m ·x) I
)=1
£
< 2L
(12)
holds for xe Ill.
Substituting (12) into the inequality (11), we
conclude that there exist N,q,9ie R, Yie Rn, such that
N
I f(x) - LCicr(Yi·X+9D I <e
i=1
(13)
is true for all xe In.
Q.E.D.
x>O
x=O
x<O
(14)
which is clearly a sigmoidal function. But the set
LCicr(YiX+9i) is not dense in C[O,I]. Therefore, we
conclude that the boundedness of cr(x) is an essential
assumption for the validity of Theorem 2.
Remark 3. If cr(x) is a monotone sigmoidal
function, then it is clear that O~cr(x)~1. As a consequence
of Theorem 2, we have that if cr(x) is a monotone sigmoidal
function, then the finite linear combinations Lqcr(Yi·X+9i)
are dense in C(In).
have
Theorem 3. If cr(x) is a bounded sigmoidal
function, then the set of finite linear combinations
N
LCicr(Yi·X+9i) is dense in LP(In) (I~p~oo). (See [1]).
i=1
For unbounded sigmoidal functions, we can prove
Theorem 3'.
If a
function
cr(x)e LP[a,b] (1~p~oo) for every pair of a,be R, a<b, then
sigmoidal
the set of finite linear combinations LCicr(Yi·X+9i) is
i=1
N
In fact, what we need to prove is that, under the
assumption of Theorem 3', the Corollary of Theorem 1 is
true in LP[a,b]' instead of in C[a,b]. The details are omitted.
Remark 1. It is clear in the proofs of Theorems
1 and 2 that the continuity of cr(x) is unnecessary for the
Remark 4. If co(x) is a measurable function and
co(x)eL [a,b] (1~p~oo), for every a,beR, a<b, and the set of
P
第 5 页
finite linear combinations LCiro(~iX+Tli) is dense in any
L P [a,b], then the set of finite linear combinations
LCiro(Yi'X+9D is dense in LP (lln). (Note a special case of
this is when ro(X)E C[a,b].) This is a mild condition
satisfied by many ro(x). Among those are many functions
that frequently occur in approximation theory, such as the
well known Schoenberg Cardinal Splines, B-Splines and
many others, including those functions satisfying Wiener
Tauberian conditions. For proof, we can follow the same
Cybenko's Approximation Theorem
167
I f(x,y) - f(-M,y)
N1
- L[f(Xi,y)-f(Xi-l,y)] cr(Kl(X-ti_l» I < ~
(15)
i=l
2
holds for all (X,Y)E [-00,00] .
Writing Ci(y)=f(Xi,y)-f(Xi-l,y), i=I, ... ,Nl,
Co(y)=f(-M,y). Repeating the argument as in the proof of
Theorem 1, there holds
line developed in the proof of Theorem 2, approximating
exp(i (mlxl+"'+mnxn» by the finite linear combinations
LCjro(A.j(mlxl+· .. +mnxn)+9j) and using Fourier series
hence
N1 N2
development.
We now give another kind of generalization.
First, we will call [-00,00] the extended real line,
with (-co,A) as the neighborhood of the point -00, and
(A,+oo) the neighborhood of +00. Thus [-00,00] is a compact
space, and [_oo,oo]n , being the product of compact spaces, is
also compact.
Similar as before, we denote C[-oo,oot the set of
continuous functions on the compact set [-co,oo]n.
With this notation, we can state Theorem 1 as .. the
set of finite linear combinations LCicr(YiX+9i) is dense in
C[-oo,oo] ".
Theorem 4. If cr(x) is a bounded sigmoidal
function, then the set of finite linear combinations
I f(x,y) - '" ~ coo cr(ro.y+9.) cr(~,y+T1') 1<£
(17)
1
1
J
J
~ IJ
i=l
J=
Corollary 1. If cr(x) is a bounded sigmoidal
function, then the set of finite linear combinations
Q.E.D.
is dense in C[O,It.
many systems, there holds similar theorem:
Remark 5. Besides sigmoidal functions, for
If roi(X),
i=O,±I, ... , is a system of functions, and if LCiroi(X) are
dense in C[O,l], then L\ ..... ~ roil (Xl)"'ro~ (Xn) are dense
in C[O,l]n, if only roi(X) satisfies very general conditions.
For example, {exp(z1mx)} +00
is such a system. In this
k=-oo
case roo (x1)···ro. (x ) = ro(i1x1+ .. ·+inxn). This is just
11
In n
is dense in C[-co,oot.
Proof. Without loss of generality, we assume
n=2. Let f(X,Y)E C[_00,00]2, then f(x,y), fixing y, is
uniformly continuous on the compact set [-00,00]. By
Theorem 1: for any £>0, there exist Nl,Kl such that
the reason why we used i m.x as the medium in the proof
of Theorem 2. Of course we can also use e
medium instead.
m·x
as the第 6 页
168
T. Chen, H. Chen and R. Liu
[1]
[2]
[3]
[4]
[5]
[6]
[7]
References
G.Cybenko, "Approximation by Superpositions of
a Sigmoidal Function", Mathematics of Control,
Signals and Systems, V.2, No.4 (1989), P.303-
314.
A.N.Komogorov, "On the Representation of
Continuous Functions of Several Variables by
Superposition of Continuous Functions of One
Variable and Addition", Dokl. Akad. Nawk. SSSR,
114, 1957 (pp.953-956) English translation,
American Math. Soc. Transl. (2), 28 (1963),
pp.55-59, MR.22#2669; 27#3760.
E.M.Stein and Guido Weiss, "Introduction to
Fourier Analysis on Euclidean Spaces", Princeton
University Press (1971).
L.K.Jones, "Constructive Approximation for
Neural Networks by Sigmoidal Functions",
Technical Report Series, No.7, Dept. of
Mathematics, University of Lowell (1988).
S.M.Carrol and B.W.Dickinson, "Construction of
Neural Nets Using Radon Transform", Preprint
1989.
K.Funahashi, "On the Approximate Realization of
Continuous Mappings by Neural Networks",
Journal of International Neural Networks (to
appear).
K.Homik, M.Stinchcombe and H.White, "Multi
layer Feedforward Networks are Universal
Approximators", Preprint 1988.