The Proxies service provides an easy way to establish secure tunnels between
your clients, which helps your applications running in different locations
to communicate with each other.
The Proxy service component relationship.
The main problem with establishing connections between two clients
in different locations on the internet is that it’s not always easy to
reach out to the other client through firewalls and NATs in between. Most
home networks are protected by NATs and you usually have to implement
a VPN network to set up a peer-to-peer connection.
The Proxies service provides clients with a meet-up endpoint for clients
to establish client-initiated connections and stitches two connections to
make an end-to-end connection between two clients. Because all connections
are client-initiated, NAT and firewalls won’t cause any connection problems.
While the Proxies service provides only a framework to establish connections
between clients, you can implement your own use cases on top of that such as
tunnels for remote commands, which is how the Devices service utilizes the
Proxies service, and TCP port forwarding just like the ZTNA service work with
the Proxies service.
A Tunnel is an end-to-end connection between two clients. It is built
with two client-initiated gRPC sessions to our meet-up endpoint on the
cloud to avoid problems caused by firewalls and NATs. As a tunnel works
as a bare data stream just like a normal TCP connection, you can implement
any protocol on top of it.
Tunnel Brokers
A tunnel broker is a part of the service backend, it provides
a mechanism for establishing tunnels between
consumers and providers.
Tunnel Consumers
A tunnel consumer is one of the end clients that connects with
the broker to establish a tunnel
with the selected provider.
Tunnel Providers
A tunnel provider is connecting to a broker
(registration), and then passively waits for consumers
to connect to it.
Code Samples
Ping/Pong over Tunnel
Let’s create a simple ping/pong service where consumers send Ping messages,
and then Providers return Pong messages.
First, define a struct that implements provider.ServiceFactory.
importprovider"github.com/cloudwan/edgelq/proxies/provider/v1"typePingServicestruct {
closedchanstruct{} // will be closed when Close() is called
calledchanstruct{}
}
// Start is called when a new tunnel is established.
// You can initialize your service here, but
func (s*PingService) Start(ctxcontext.Context) error {
s.closed = make(chanstruct{}, 0)
s.called = make(chanstruct{}, 1)
}
// Send is called when the Consumer sends data to the Consumer.
func (s*PingService) Send(data []byte) error {
if string(data) !="PING" {
returnfmt.Errorf("unexpected message received")
}
select {
cases.called<-stcut{}{}:
// asked to Recv() to send Pong
case<-s.closed:
it'sbeingclosed, giveup }
returnnil}
// Recv is called to receive message from the service.
func (s*PingService) Recv() ([]byte, error) {
select {
case<-s.called:
return []byte("PONG"), nilcase<-s.closed:
returnnil, fmt.Error("service is closing")
}
}
// Close is called when the connection is going to be closed.
func (s*PingService) Close() error {
close(s.closed)
}
Then, create a Provider with the service.
import (
broker"github.com/cloudwan/edgelq/proxies/client/v1/broker"provider"github.com/cloudwan/edgelq/proxies/provider/v1"project"github.com/cloudwan/edgelq/proxies/resources/v1/project")
varclientproxcbroker.BrokerServiceClient// setup a client
// List of services (we have only one at this moment).
services:=map[string]provider.ServiceFactory{
"ping": func(ctxcontext.Context, arg []byte) (provider.Service, error) {
returnNewPingService(), nil },
}
// Create a Provider
provier:=provider.NewProvider(
project.NewNameBuilder().SetId("YourProjectID").Name(),
"my-first-provider-id",
"us-west2",
"ping.edgelq.com",
services,
)
// Run it.
provider.Run(ctx, client)
Now your Ping service is “Provided”.
Next, let’s consume the service as a Consumer.
import (
broker"github.com/cloudwan/edgelq/proxies/client/v1/broker")
varclientproxcbroker.BrokerServiceClient// setup a client
// define an “Outlet” that implements Recv() and Send()
typePingOutletstruct {
sentbool}
// Send writes data from the other end to the Outlet.
func (o*PingOutlet) Send(data []byte) error {
fmt.Printf(“Received: %s\n”, string(data))
}
// Recv outputs data from the Outlet
func (o*PingOutlet) Recv() ([]byte, error) {
ifo.sent {
Returnnil, io.EOF }
o.sent = truereturn []byte(“PING”), nil}
err:=consumer.Connect(
context.Background(),
client,
proxrproject.NewNameBuilder().SetId("YourProjectID").Name(),
"my-first-consumer-id",
"us-west2",
"ping.edgelq.com",
"my-first-provider-id", // provider ID to connect to
"ping", // service name
nil, // no initial arg
PingOutlet{},
)
1 -
Understanding the proxies.edgelq.com service APIv1, in proto package ntt.proxies.v1.
Service proxies.edgelq.com in version v1, proto package ntt.proxies.v1
Here is the list of resources supported in Proxies service APIv1:
Name of Project When creating a new instance, this field is optional and if not provided, it will be generated automatically. Last ID segment must conform to the following regex: [a-z][a-z0-9\-]{0,28}[a-z0-9]
Metadata is an object with information like create, update and delete time (for async deleted resources), has user labels/annotations, sharding information, multi-region syncing information and may have non-schema owners (useful for taking ownership of resources belonging to lower level services by higher ones).
Filter - filter results by field criteria. Simplified SQL-like syntax with following operators: <=, >=, =, !=, <, >, LIKE, CONTAINS (aliases CONTAIN, HAS, HAVE), IN, IS [NOT] NULL
field_mask
.google.protobuf.FieldMask
A list of extra fields to be obtained for each response item on top of fields defined by request field view
A token to retrieve next page of results. Pass this value in the ListProjectsRequest.page_token.
current_offset
int32
Current offset from the first page or 0 if no page tokens were given, paging info was not requested or there was an error while trying to get it). Page index can be computed from offset and limit provided in a request.
total_results_count
int32
Number of total Projects across all pages or 0, if there are no items, paging info was not requested or there was an error while trying to get it.
Type of a watch. Identifies how server stream data to a client, which fields in a request are allowed and which fields in response are relevant.
page_size
int32
Requested page size. Server may return fewer Projects than requested. If unspecified, server will pick an appropriate default. Can be populated only for stateful watch type.
A token identifying watch resume point from previous session. Can be populated only for stateless watch type.
starting_time
.google.protobuf.Timestamp
Point in the time from which we want to start getting updates. This field can be populated only for stateless watch type and if resume token is not known yet. If specified, initial snapshot will NOT be provided. It is assumed client can obtain it using separate means. Watch responses will contain resume tokens which should be used to resume broken connection.
Filter - filter results by field criteria. Simplified SQL-like syntax with following operators: <=, >=, =, !=, <, >, LIKE, CONTAINS (aliases CONTAIN, HAS, HAVE), IN, IS [NOT] NULL
field_mask
.google.protobuf.FieldMask
A list of extra fields to be obtained for each response item on top of fields defined by request field view Changes to Project that don’t affect any of masked fields won’t be sent back.
View defines list of standard response fields present in response items. Additional fields can be amended by request field field_mask Changes to Project that don’t affect any of masked fields won’t be sent back.
max_chunk_size
int32
Maximum amount of changes in each response message. Query result response is divided on the server side into chunks with size of a specified amount to limit memory footprint of each message. Responses will hold information whether more elements will continue for the actual change. If unspecified, server will pick an appropriate default.
If request specified max_chunk_size (or this limit was enforced if stateless watch has been chosen), then responses with “full changeset” will be divided into chunks. Client should keep receiving messages and, once is_current has value true, combine this recent message with all previous ones where is_current is false. If this is the first is_current in a whole watch stream, then it means that client should have, at this moment, contain snapshot of the current situation (or more accurately, snapshot of situation at the moment of request). All Projects will be of type Added/Current (depending on watch_type specified in the request). Further responses will be incremental - however messages may still be chunked and is_current logic still applies. is_current is always true for stateful watch if max_chunk_size was left to 0.
When present, PageTokens used for page navigation should be updated. Present only if is_current is true (last chunk).
resume_token
string
Token that can be used if current connection drops and client needs to reconnect. Populated only for stateless watch type. Present only if is_current is true (last chunk).
snapshot_size
int64
Server may occasionally send information how many resources should client have in its state so far (response message without any changes, but with snapshot_size field specified). If client has different value than the one sent by the server, then it should be treated by a client as an error and should reconnect. If value is smaller then 0, then client should ignore this field as unpopulated. This field should be checked only for stateless watch. In stateful those kind of errors are handled by the server side. Will be never sent together with is_current, is_soft_reset and is_hard_reset flags.
is_soft_reset
bool
In case of internal issue server may send response message with this flag. It indicates that client should drop all changes from recent responses where is_current is false only! If last message had is_current set to true, client should do nothing and process normally. Resume token received before is still valid. This field should be checked only for stateless watch. In stateful those kind of errors are handled by the server side. Will never be sent along with is_current, is_hard_reset or snapshot_size.
is_hard_reset
bool
In case of internal issue server may send response message with this flag. After receiving, client should clear whole state (drop all changes received so far) as server will send new snapshot (Projects will contains changes of type Current only). Any resume tokens should be discarded as well. This field should be checked only for stateless watch. In stateful those kind of errors are handled by the server side. Will never be sent along with is_current, is_soft_reset or snapshot_size.
Conditional update applied to request if update should be executed only for specific resource state. If this field is populated, then server will fetch existing resource, compare with the one stored in the cas field (after applying field mask) and proceed with update only and only if they match. Otherwise RPC error Aborted will be returned.
Conditional desired state of a resource before update.
field_mask
.google.protobuf.FieldMask
Field paths from conditional state of resource server should check and compare.
UpdateProjectRequest.ResponseMask Message
ResponseMask allows client to reduce response message size.
Name
Type
Description
skip_entire_response_body
bool
If this flag has value true, then response will contain just empty resource without any fields populated. Field body_mask is ignored if set.
updated_fields_only
bool
Include all fields that were actually updated during processing. Note this may be larger than update mask if some fields were computed additionally. Name is added as well.
body_mask
.google.protobuf.FieldMask
If this field is populated, then resource in response will contain only specific fields. If skip_entire_response_body is true, this field is ignored.
New version of Project or masked difference, depending on mask_changes instrumentation of issued [WatchProjectRequest] or [WatchProjectsRequest]
field_mask
.google.protobuf.FieldMask
Used when mask_changes is set, contains field paths of modified properties.
previous_view_index
int32
Previous view index specifies previous position of modified Project. When modification doesn’t affect sorted order, value will remain identical to [view_index].
view_index
int32
Integer specifying Project new index in resulting query view.
ProjectChange.Removed Message
Removed is returned when Project is deleted or leaves Query view
Name of Project When creating a new instance, this field is optional and if not provided, it will be generated automatically. Last ID segment must conform to the following regex: [a-z][a-z0-9-]{0,28}[a-z0-9]
Filter - filter results by field criteria. Simplified SQL-like syntax with following operators: <=, >=, =, !=, <, >, LIKE, CONTAINS (aliases CONTAIN, HAS, HAVE), IN, IS [NOT] NULL
field_mask
.google.protobuf.FieldMask
A list of extra fields to be obtained for each response item on top of fields defined by request field view
A token to retrieve next page of results. Pass this value in the ListProjectsRequest.page_token.
current_offset
int32
Current offset from the first page or 0 if no page tokens were given, paging info was not requested or there was an error while trying to get it). Page index can be computed from offset and limit provided in a request.
total_results_count
int32
Number of total Projects across all pages or 0, if there are no items, paging info was not requested or there was an error while trying to get it.
Type of a watch. Identifies how server stream data to a client, which fields in a request are allowed and which fields in response are relevant.
page_size
int32
Requested page size. Server may return fewer Projects than requested. If unspecified, server will pick an appropriate default. Can be populated only for stateful watch type.
A token identifying watch resume point from previous session. Can be populated only for stateless watch type.
starting_time
.google.protobuf.Timestamp
Point in the time from which we want to start getting updates. This field can be populated only for stateless watch type and if resume token is not known yet. If specified, initial snapshot will NOT be provided. It is assumed client can obtain it using separate means. Watch responses will contain resume tokens which should be used to resume broken connection.
Filter - filter results by field criteria. Simplified SQL-like syntax with following operators: <=, >=, =, !=, <, >, LIKE, CONTAINS (aliases CONTAIN, HAS, HAVE), IN, IS [NOT] NULL
field_mask
.google.protobuf.FieldMask
A list of extra fields to be obtained for each response item on top of fields defined by request field view Changes to Project that don’t affect any of masked fields won’t be sent back.
View defines list of standard response fields present in response items. Additional fields can be amended by request field field_mask Changes to Project that don’t affect any of masked fields won’t be sent back.
max_chunk_size
int32
Maximum amount of changes in each response message. Query result response is divided on the server side into chunks with size of a specified amount to limit memory footprint of each message. Responses will hold information whether more elements will continue for the actual change. If unspecified, server will pick an appropriate default.
If request specified max_chunk_size (or this limit was enforced if stateless watch has been chosen), then responses with “full changeset” will be divided into chunks. Client should keep receiving messages and, once is_current has value true, combine this recent message with all previous ones where is_current is false. If this is the first is_current in a whole watch stream, then it means that client should have, at this moment, contain snapshot of the current situation (or more accurately, snapshot of situation at the moment of request). All Projects will be of type Added/Current (depending on watch_type specified in the request). Further responses will be incremental - however messages may still be chunked and is_current logic still applies. is_current is always true for stateful watch if max_chunk_size was left to 0.
When present, PageTokens used for page navigation should be updated. Present only if is_current is true (last chunk).
resume_token
string
Token that can be used if current connection drops and client needs to reconnect. Populated only for stateless watch type. Present only if is_current is true (last chunk).
snapshot_size
int64
Server may occasionally send information how many resources should client have in its state so far (response message without any changes, but with snapshot_size field specified). If client has different value than the one sent by the server, then it should be treated by a client as an error and should reconnect. If value is smaller then 0, then client should ignore this field as unpopulated. This field should be checked only for stateless watch. In stateful those kind of errors are handled by the server side. Will be never sent together with is_current, is_soft_reset and is_hard_reset flags.
is_soft_reset
bool
In case of internal issue server may send response message with this flag. It indicates that client should drop all changes from recent responses where is_current is false only! If last message had is_current set to true, client should do nothing and process normally. Resume token received before is still valid. This field should be checked only for stateless watch. In stateful those kind of errors are handled by the server side. Will never be sent along with is_current, is_hard_reset or snapshot_size.
is_hard_reset
bool
In case of internal issue server may send response message with this flag. After receiving, client should clear whole state (drop all changes received so far) as server will send new snapshot (Projects will contains changes of type Current only). Any resume tokens should be discarded as well. This field should be checked only for stateless watch. In stateful those kind of errors are handled by the server side. Will never be sent along with is_current, is_soft_reset or snapshot_size.
Conditional update applied to request if update should be executed only for specific resource state. If this field is populated, then server will fetch existing resource, compare with the one stored in the cas field (after applying field mask) and proceed with update only and only if they match. Otherwise RPC error Aborted will be returned.
Conditional desired state of a resource before update.
field_mask
.google.protobuf.FieldMask
Field paths from conditional state of resource server should check and compare.
UpdateProjectRequest.ResponseMask Message
ResponseMask allows client to reduce response message size.
Name
Type
Description
skip_entire_response_body
bool
If this flag has value true, then response will contain just empty resource without any fields populated. Field body_mask is ignored if set.
updated_fields_only
bool
Include all fields that were actually updated during processing. Note this may be larger than update mask if some fields were computed additionally. Name is added as well.
body_mask
.google.protobuf.FieldMask
If this field is populated, then resource in response will contain only specific fields. If skip_entire_response_body is true, this field is ignored.
New version of Project or masked difference, depending on mask_changes instrumentation of issued [WatchProjectRequest] or [WatchProjectsRequest]
field_mask
.google.protobuf.FieldMask
Used when mask_changes is set, contains field paths of modified properties.
previous_view_index
int32
Previous view index specifies previous position of modified Project. When modification doesn’t affect sorted order, value will remain identical to [view_index].
view_index
int32
Integer specifying Project new index in resulting query view.
ProjectChange.Removed Message
Removed is returned when Project is deleted or leaves Query view
