[boost] [beast] Chunking example

[Bjorn Reese via Boost]

How would an example of receiving a chunked transfer using the
synchronous API look? In pseudo-code, I want to do as follows:

establish connection
while not closed
read chunk
print chunk

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

[Vinnie Falco via Boost]

On Sun, Jul 2, 2017 at 5:41 AM, Bjorn Reese via Boost

Post by Bjorn Reese via Boost
How would an example of receiving a chunked transfer using the
establish connection
while not closed
read chunk
print chunk

That's going to be be quite involved, you would need to subclass
beast::http::basic_parser and implement on_chunk to remember the chunk
size. It is not something that I recommend nor is it a common
use-case. HTTP applications are not supposed to care about the
boundaries between chunks since intermediates like proxies are allowed
to re-frame chunked message payloads. However, some applications may
wish to decode the chunk-extension and Beast handles that, but you
have to subclass beast::http::basic_parser for it.

Its possible that what you are really asking is how to read a message
payload incrementally? One of the examples performs a similar
operation:
<http://vinniefalco.github.io/beast/beast/more_examples/http_relay.html>

Something like this should achieve your goal (note, untested):

/* This function reads a message using a fixed size buffer to hold
portions of the body, and prints the body contents to a `std::ostream`.
*/
template<
bool isRequest,
class SyncReadStream,
class DynamicBuffer>
void
read_and_print_body(
std::ostream& os,
SyncReadStream& stream,
DynamicBuffer& buffer,
error_code& ec)
{
parser<isRequest, buffer_body> p;
read_header(stream, buffer, p, ec);
if(ec)
return;
while(! p.is_done())
{
char buf[512];
p.get().body.data = buf;
p.get().body.size = sizeof(buf);
read(stream, buffer, p, ec);
if(ec == error::need_buffer)
ec.assign(0, ec.category());
if(ec)
return;
os.write(buf, sizeof(buf) - p.get().body.size);
}
}

Thanks

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

[Peter Dimov via Boost]

Post by Vinnie Falco via Boost
Its possible that what you are really asking is how to read a message
<http://vinniefalco.github.io/beast/beast/more_examples/http_relay.html>

This preempts one of my questions, I was going to ask for precisely that
example, as it's referenced in the documentation as a use case here:

http://vinniefalco.github.io/stage/beast/review/beast/using_http/parser_stream_operations.html

as

"* Receive a large body using a fixed-size buffer."


_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

[Bjorn Reese via Boost]

Post by Vinnie Falco via Boost
Its possible that what you are really asking is how to read a message
payload incrementally? One of the examples performs a similar

I am trying to investigate how your message model handles trailing
headers.

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

[Vinnie Falco via Boost]

On Sun, Jul 2, 2017 at 6:33 AM, Bjorn Reese via Boost

Post by Bjorn Reese via Boost
I am trying to investigate how your message model handles trailing
headers.

Trailers are simply inserted into the message fields when they are received.

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

[Bjorn Reese via Boost]

Post by Vinnie Falco via Boost
Trailers are simply inserted into the message fields when they are received.

Can I differentiate between headers and trailers, or are they mixed
together?

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

[Vinnie Falco via Boost]

On Sun, Jul 2, 2017 at 6:52 AM, Bjorn Reese via Boost

Can I differentiate between headers and trailers, or are they mixed together?

They are mixed together. You can look at the "Trailer" field after
receiving the header to know which trailers have been promised. And
then when the message is complete, inspect the headers again looking
for those specific fields.

If you absolutely need to distinguish them, for example to perform
more robust error checking, reject trailers which were not promised or
which are disallowed as per rfc7230, then you can subclass
basic_parser and provide a suitable implementation of on_field. In
this case you could store them in a separate container if desired.

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

[Bjorn Reese via Boost]

Post by Vinnie Falco via Boost
to re-frame chunked message payloads. However, some applications may
wish to decode the chunk-extension and Beast handles that, but you
have to subclass beast::http::basic_parser for it.

So chunk-ext fields are not passed to the user unless they write a
parser to extract them?

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

[Vinnie Falco via Boost]

On Sun, Jul 2, 2017 at 7:34 AM, Bjorn Reese via Boost

Post by Bjorn Reese via Boost
So chunk-ext fields are not passed to the user unless they write a
parser to extract them?

If you want the chunk extensions you have to subclass basic_parser
(not the same as writing a parser). Here's an example of what a
user-defined subclass could start with:

template<bool isRequest, class Body,
class Allocator = std::allocator<char>>
class parser
: public basic_parser<isRequest,
parser<isRequest, Body, Allocator>>
{
using base_type = basic_parser<isRequest,
parser<isRequest, Body, Allocator>>;

message<isRequest, Body, basic_fields<Allocator>> m_;
boost::optional<typename Body::writer> wr_;

public:
using value_type =
message<isRequest, Body, basic_fields<Allocator>>;

parser() = default;
parser(parser const&) = delete;
parser(parser&& other) = default;
parser& operator=(parser const&) = delete;

template<class... Args>
explicit
parser(Args&&... args)
: m_(std::forward<Args>(args))
{
}

value_type const&
get() const { return m_; }
value_type& get() { return m_; }
value_type release() { return std::move(m_); }

private:
friend class basic_parser<isRequest, parser>;

void
on_request(verb method, string_view method_str,
string_view target, int version, error_code& ec)
{
try
{
m_.target(target);
if(method != verb::unknown)
m_.method(method);
else
m_.method_string(method_str);
ec.assign(0, ec.category());
}
catch(std::bad_alloc const&)
{
ec = error::bad_alloc;
}
m_.version = version;
}

void
on_response(int code,
string_view reason,
int version, error_code& ec)
{
m_.result(code);
m_.version = version;
try
{
m_.reason(reason);
ec.assign(0, ec.category());
}
catch(std::bad_alloc const&)
{
ec = error::bad_alloc;
}
}

void
on_field(field name, string_view name_string,
string_view value, error_code& ec)
{
try
{
m_.insert(name, name_string, value);
ec.assign(0, ec.category());
}
catch(std::bad_alloc const&)
{
ec = error::bad_alloc;
}
}

void
on_header(error_code& ec)
{
ec.assign(0, ec.category());
}

void
on_body(boost::optional<
std::uint64_t> const& content_length,
error_code& ec)
{
wr_.emplace(m_, content_length, ec);
}

std::size_t
on_data(string_view s, error_code& ec)
{
return wr_->put(boost::asio::buffer(
s.data(), s.size()), ec);
}

void
on_chunk(std::uint64_t size,
string_view ext, error_code& ec)
{
ec.assign(0, ec.category());
}

void
on_complete(error_code& ec)
{
if(wr_)
wr_->finish(ec);
else
ec.assign(0, ec.category());
}
};

The function on_chunk() is called for each new chunk header, and gets
passed the size of the chunk as well as the extension.

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

[Bjorn Reese via Boost]

Post by Vinnie Falco via Boost
If you want the chunk extensions you have to subclass basic_parser
(not the same as writing a parser). Here's an example of what a

Chunking is (abstractly speaking) similiar to WebSocket frames, so I
am surprised at how different these solutions are.

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

[Vinnie Falco via Boost]

On Sun, Jul 2, 2017 at 9:18 AM, Bjorn Reese via Boost

Post by Bjorn Reese via Boost
Chunking is (abstractly speaking) similiar to WebSocket frames, so I
am surprised at how different these solutions are.

Actually Beast treats them very similarly. Its not generally possible
in Beast to see the WebSocket frame boundaries either (especially if
you have compression on). It might look like you can see frame
boundaries because of the function `websocket::stream::read_frame` but
that function can cross boundaries or return less than a full frame.

As per rfc6455 (WebSocket) interfaces presented to the application
(`beast::websocket::stream` in this case) are free to reframe
messages.

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

[Bjorn Reese via Boost]

Post by Vinnie Falco via Boost
Actually Beast treats them very similarly. Its not generally possible

But their APIs are quite different.

Post by Vinnie Falco via Boost
in Beast to see the WebSocket frame boundaries either (especially if
you have compression on). It might look like you can see frame
boundaries because of the function `websocket::stream::read_frame` but
that function can cross boundaries or return less than a full frame.
As per rfc6455 (WebSocket) interfaces presented to the application
(`beast::websocket::stream` in this case) are free to reframe
messages.

Does Beast re-packet WebSocket frames or HTTP chunks?

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

[Vinnie Falco via Boost]

On Sun, Jul 2, 2017 at 9:49 AM, Bjorn Reese via Boost

Post by Bjorn Reese via Boost
But their APIs are quite different.

I'm not sure I agree that the claimed abstract similarity of the
concept warrants efforts to make the interfaces "consistent." The APIs
were not planned ahead of time, they were arrived at after months of
reworking to handle all of the use cases. Some use-cases were made
very easy, at the expense of some other less common use-cases
requiring more effort (such as extracting chunk extensions).

Post by Bjorn Reese via Boost
Does Beast re-packet WebSocket frames...

Yes

Post by Bjorn Reese via Boost
or HTTP chunks?

Chunk-encoding is removed from incoming messages before presentation
to the application layer. For outgoing messages, Beast is under
control over how pieces are chunked. However, the Body::reader type
has influence but not complete control (the specifications is
intentionally vague).

Users can set chunk extensions and trailers by using a ChunkDecorator
http://vinniefalco.github.io/beast/beast/using_http/serializer_stream_operations.html#beast.using_http.serializer_stream_operations.chunk_decorators

Thanks

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

[Bjorn Reese via Boost]

Post by Vinnie Falco via Boost
On Sun, Jul 2, 2017 at 9:49 AM, Bjorn Reese via Boost

Post by Bjorn Reese via Boost
Does Beast re-packet WebSocket frames...

Yes

Under what conditions?

The reason I ask is that Beast claims to provide a low-level API, so I
would assume that users can have complete control over this.

Post by Vinnie Falco via Boost

Post by Bjorn Reese via Boost
or HTTP chunks?

Chunk-encoding is removed from incoming messages before presentation
to the application layer. For outgoing messages, Beast is under

Does Beast concatenate chunks before passing them to the application
layer, or does the application layer get the chunks in the same way as
they were received?

Post by Vinnie Falco via Boost
control over how pieces are chunked. However, the Body::reader type
has influence but not complete control (the specifications is
intentionally vague).

Why does the user not have complete control? Under what conditions will
Beast re-packet outgoing chunks?

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

[Vinnie Falco via Boost]

On Sun, Jul 2, 2017 at 10:51 AM, Bjorn Reese via Boost

Post by Bjorn Reese via Boost

Post by Bjorn Reese via Boost
Does Beast re-packet WebSocket frames...

...
Yes

...
Under what conditions?

There are stream settings for auto-fragmentation of messages into
frames of prescribed sizes. Also, when the permessage-deflate
algorithm is enabled, Beast reframes messages to fit in the fixed
compression and decompression buffers it uses.

Post by Bjorn Reese via Boost
The reason I ask is that Beast claims to provide a low-level API, so I
would assume that users can have complete control over this.

I don't think that's a good assumption. From
<https://tools.ietf.org/html/rfc6455#section-1.2>

"The WebSocket message does not necessarily correspond to a particular
network layer framing, as a fragmented message may be coalesced or
split by an intermediary."

beast::websocket::stream is considered the first intermediary, and it
exercises its privilege to reframe message octets. This is established
practice, Autobhan websocket implementations also do this; they have a
similar auto-fragment feature (which I copied).

Post by Bjorn Reese via Boost

Post by Bjorn Reese via Boost
or HTTP chunks?

The "low-level" claims refer to the implementation of the HTTP
protocol. It doesn't imply that callers will have control over chunks.
Intermediaries along the HTTP path are allowed to rechunk message
payloads.

Perhaps I have overlooked something. What is the use-case for
application level control over chunking?

Post by Bjorn Reese via Boost
Does Beast concatenate chunks before passing them to the application
layer, or does the application layer get the chunks in the same way as
they were received?

The application sees the message body only after the chunked
Transfer-Encoding has been removed.

Post by Bjorn Reese via Boost
Why does the user not have complete control? Under what conditions will
Beast re-packet outgoing chunks?

Beast's implementation does not currently reframe serialized chunks,
but there is no specification for how Beast will perform the chunked
encoding. This could be changed in the future, if there was a
compelling use-case.

Thanks

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

[Bjorn Reese via Boost]

Post by Vinnie Falco via Boost
Perhaps I have overlooked something. What is the use-case for
application level control over chunking?

The publish-subscribe pattern. Consider a device (e.g. a sensor), that
publishes events (e.g. measurements) that are distributed unevenly
and unpredictably over time. Each chunk contains a single event, so
preserving the chunk boundary is important.

Post by Vinnie Falco via Boost
The application sees the message body only after the chunked
Transfer-Encoding has been removed.

Can you elaborate with an example?

Post by Vinnie Falco via Boost
Beast's implementation does not currently reframe serialized chunks,
but there is no specification for how Beast will perform the chunked
encoding. This could be changed in the future, if there was a
compelling use-case.

Beast should not break the end-to-end principle unless explicitly
allowed by the application.

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

[Vinnie Falco via Boost]

On Tue, Jul 4, 2017 at 10:49 AM, Bjorn Reese via Boost

Post by Bjorn Reese via Boost
That was scary reading, because HTTPbis seemed to be unaware that such
deprecation would break some of IETF's own standards, e.g. RFC 3507.

I agree :) That's why I made sure that Beast is capable of sending and
receiving chunk-extensions.

Post by Bjorn Reese via Boost
Another use-case that is used in practice is for in-band meta-data.
... chunk extensions are used to carry the meta-data.

That makes sense, and that's a supported use-case. You can send the
extensions and you can receive them.

Post by Bjorn Reese via Boost
Consider a device (e.g. a sensor), that
publishes events (e.g. measurements) that are distributed unevenly
and unpredictably over time. Each chunk contains a single event, so
preserving the chunk boundary is important.

You're talking about a "perpetual message" of some kind, where the end
of the message never arrives. I think that's kind of an abuse of HTTP.
Isn't Websocket more suited for that? It supports message boundaries
which are not rewritten by intermediates (unlike HTTP).

With respect to detecting chunk boundaries on input, this is possible,
but... I can't guarantee it on output. In fact I'm not even sure how
you would convince Beast to keep a message in "perpetually sending"
mode since that is a use-case not anticipated by rfc7230.

I think the sane way to do what you want is to use Beast for sending
the header, which works great, and then take responsibility for
outputting the body chunks yourself. True, you'll have to be getting
your hands dirty with the protocol details of HTTP but that's already
assumed because you 1. want to know about chunks, 2. want to control
their output, 3. want to send an infinitely long message. I'm not
losing sleep that Beast doesn't make this use-case ridiculously easy.

Post by Bjorn Reese via Boost

Post by Vinnie Falco via Boost
The application sees the message body only after the chunked
Transfer-Encoding has been removed.

...
Can you elaborate with an example?

Let me rephrase that, beast::http::parser appends chunk bodies into
the body container stored in the message (std::string for example).
From the caller's perspective they just see that the message body is
growing. There's no guarantee that each increment of growth exactly
corresponds to one chunk.

However, if you want to see chunks that is possible by subclassing
basic_parser and handling it in the on_chunk callback, which is called
after the chunk header is received and gets passed the size of the
upcoming body as well as the chunk-extension.

I suppose I could easily add an optional user-facing callback to
beast::http::parser with the signature <void(std::uint64_t,
string_view, error_code&> to allow users to easily detect chunk
boundaries during parsing. It would be a handful of lines of code
added here and there (similar to the on_header callback already
present):
<https://github.com/vinniefalco/Beast/blob/78a065ba39836d91d7e70d93de7f9140f518083b/include/beast/http/parser.hpp#L298>

I haven't done it because I don't like to design theoretical features
which are unlikely to be used. I have found that when I do such
things, especially for features that I have little experience with
(like chunk extensions) I end up discovering later that the interfaces
don't satisfy real-world needs and then have to be changed.

Post by Bjorn Reese via Boost
Beast should not break the end-to-end principle unless explicitly
allowed by the application.

I think if users demand control over outgoing chunks, its reasonable
to ask that they manually output the chunked message body; Beast will
take care of the header.

Post by Bjorn Reese via Boost
Is it possible to provide a HTTP 200 response alias for Beast?
...
This is needed for SHOUTcast internet radio servers.

There are no provisions for this. Beast's parser is strict. Responses
which don't start with a valid HTTP-version generate an error.

That said, I recognize that sometimes in the real world if you want to
get in to the club you have to pay the cover. I can add a simple
customization point to basic_parser giving the derived class the
opportunity to parse the status-line if it wants, to handle this case.

However, it won't be for free - I'll kindly ask that the stakeholders
(the people who intend to use the feature) participate in the design
and code review in order that the result suits their needs. Its not my
area of knowledge so if we want to get this right, those with more
experience should weigh in.

And by the way, thanks for pointing this out! This is exactly the sort
of thing that Beast needs in order to mature into something great.

Post by Bjorn Reese via Boost
The chunk decorator uses boost::asio::null_buffers to overload the
field trailer.
Given your desire to adapt Beast to Networking TS, do notice that
null_buffers are not part of N4656 (they have been replaced by
dedicated wait functions.)

Good point!! That will have to be addressed when Beast is ported.
Right now the chunk decorator is pretty raw. Again, this is a case
where a feature has been created with no stakeholders to vet the
design. I wrote the minimum amount of interface necessary to make sure
Beast supports the feature. With feedback from people actually using
it in the field, it can be improved.

Thanks

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

[Gavin Lambert via Boost]

Post by Vinnie Falco via Boost
You're talking about a "perpetual message" of some kind, where the end
of the message never arrives. I think that's kind of an abuse of HTTP.
Isn't Websocket more suited for that? It supports message boundaries
which are not rewritten by intermediates (unlike HTTP).
With respect to detecting chunk boundaries on input, this is possible,
but... I can't guarantee it on output. In fact I'm not even sure how
you would convince Beast to keep a message in "perpetually sending"
mode since that is a use-case not anticipated by rfc7230.

Perpetual documents are relatively common in server push techniques,
even before chunked transfer was a thing. (multipart/x-mixed-replace
somewhat resembles chunked transfer, except with the behaviour that the
new content completely replaces the old content instead of being
appended to it.)

The other common pattern (Comet style long poll) is to accept a request
and parse the headers but then just keep the connection open
indefinitely before finally sending a response.

Granted these are both old technologies now and SSE and WebSockets are
"better", but there are still environments where the older ones work and
the newer ones don't, and examples still exist in the wild.


_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

[Vinnie Falco via Boost]

On Tue, Jul 4, 2017 at 4:31 PM, Gavin Lambert via Boost

The other common pattern (Comet style long poll) is to accept a request and
parse the headers but then just keep the connection open indefinitely before
finally sending a response.

That's a supported use case, call `write_header` or
`async_write_header` on the `serializer` and then at your leisure call
`write` or `async_write` on the `serializer` again to finish.

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

[Bjorn Reese via Boost]

Post by Vinnie Falco via Boost
With respect to detecting chunk boundaries on input, this is possible,
but... I can't guarantee it on output. In fact I'm not even sure how

Why not? When the user passes a buffer to Beast in chunked-mode, then
serialization can convert it directly into a chunk. That ought to be
very easy to do.

Post by Vinnie Falco via Boost
you would convince Beast to keep a message in "perpetually sending"
mode since that is a use-case not anticipated by rfc7230.

Are you implying that Beast has an upper limit for persistent
connections? If not, what are you trying to tell me?

The use cases I have mentioned are way older than RFC 7230 (and RFC 6455
to answer your "why not WebSocket?" question.)

Post by Vinnie Falco via Boost
I suppose I could easily add an optional user-facing callback to
beast::http::parser with the signature <void(std::uint64_t,
string_view, error_code&> to allow users to easily detect chunk
boundaries during parsing. It would be a handful of lines of code

A simpler model is to send a buffer when it is passed to Beast, and
likewise pass a chunk to the application as it is parsed. This preserves
the chunk boundaries without any special facilities.

Post by Vinnie Falco via Boost
I think if users demand control over outgoing chunks, its reasonable
to ask that they manually output the chunked message body; Beast will
take care of the header.

Does this mean that the user has to manually insert chunk-size etc?

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

[Vinnie Falco via Boost]

On Thu, Jul 6, 2017 at 12:41 PM, Bjorn Reese via Boost

Post by Bjorn Reese via Boost
When the user passes a buffer to Beast in chunked-mode, then
serialization can convert it directly into a chunk. That ought to be
very easy to do.

Something like that could be made to work if the caller is using
`buffer_body`. That type of body represents caller supplied buffers,
there could be a guarantee that each caller provided buffer will be
output as a single chunk:

<https://github.com/vinniefalco/Beast/blob/89c416cde64de3265299ced216a6eef9428c2a51/include/beast/http/buffer_body.hpp#L22>

Post by Bjorn Reese via Boost

Post by Vinnie Falco via Boost
you would convince Beast to keep a message in "perpetually sending"
mode since that is a use-case not anticipated by rfc7230.

Are you implying that Beast has an upper limit for persistent
connections? If not, what are you trying to tell me?

There's no upper limit, but I have not given consideration to a mode
where a BodyReader never runs out of buffers. Perhaps it already works
without change. But there are no tests for it and I have not tried it,
so I cannot make a claim that it works.

Post by Bjorn Reese via Boost
...and likewise pass a chunk to the application as it is parsed.
This preserves the chunk boundaries without any special facilities.

Beast does not buffer the entire chunk in memory so the application
would only see pieces of it. The callback-based solution I provided,
is the closest to what you're asking ("pass a chunk to the
application").

Post by Bjorn Reese via Boost

Post by Vinnie Falco via Boost
I think if users demand control over outgoing chunks, its reasonable
to ask that they manually output the chunked message body; Beast will
take care of the header.

...
Does this mean that the user has to manually insert chunk-size etc?

In the worst case, yes, but if there is a need for an easier way for
users to manually produce chunked output, a simple adapter can be
provided. Beast had one which I removed after I made significant
improvements to the serialization interfaces, it can be brought back.

This function takes a caller provided ConstBufferSequence and lazily
transforms it into a new ConstBufferSequence (non-allocating) which
performs a chunk-encoding:
<https://github.com/vinniefalco/Beast/blob/1e3543f63ed46bb1b2e698939f3cf0b1324dc180/include/beast/http/detail/chunk_encode.hpp#L88>

Thanks

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

[Vinnie Falco via Boost]

On Thu, Jul 6, 2017 at 12:41 PM, Bjorn Reese via Boost

...

I've thought about your feedback and I think this is an issue that is
worth addressing. Some design work needs to be done to make it fit
into the greater whole, I've opened an issue. Feel free to subscribe
and add to it, the more feedback the better the result:
<https://github.com/vinniefalco/Beast/issues/614>

Thanks!

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

[Bjorn Reese via Boost]

Post by Vinnie Falco via Boost
parser<isRequest, buffer_body> p;
read_header(stream, buffer, p, ec);
if(ec)
return;
while(! p.is_done())
{
char buf[512];
p.get().body.data = buf;
p.get().body.size = sizeof(buf);
read(stream, buffer, p, ec);

If the above is used to read a chunked transfer, what happens to
chunk-ext fields? Are they inserted as header fields or are they
lost?

Given that this is an example of incremental reading, why does it
use read() rather than read_some()?

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

[Vinnie Falco via Boost]

On Sun, Jul 2, 2017 at 10:54 AM, Bjorn Reese via Boost

Post by Bjorn Reese via Boost
If the above is used to read a chunked transfer, what happens to
chunk-ext fields? Are they inserted as header fields or are they
lost?

Chunk extensions are not valid HTTP headers. `beast::http::parser`
does not store them in the `basic_fields`. It doesn't store them at
all, they are simply discarded.

From <https://tools.ietf.org/html/rfc7230#section-4.1.1>
"The chunked encoding is specific to each connection and is likely to
be removed or recoded by each recipient (including intermediaries)
before any higher-level application would have a chance to inspect the
extensions. Hence, use of chunk extensions is generally limited to
specialized HTTP services such as "long polling" (where client and
server can have shared expectations regarding the use of chunk
extensions) or for padding within an end-to-end secured connection.

To my understanding, chunk-extensions are a rare niche use-case with
meaning only to applications using a custom interpretation at each end
of the connection. In fact 5 years ago the IETF almost deprecated
them:
<https://trac.ietf.org/trac/httpbis/ticket/343>

Beast doesn't go out of its way to help you get at the extensions, but
it also doesn't make it impossible.

On the other hand, I do not have significant expertise with HTTP
servers; if a compelling use-case presents itself this is an aspect of
the library which may be improved, in a backward-compatible way.

Post by Bjorn Reese via Boost
Given that this is an example of incremental reading, why does it
use read() rather than read_some()?

The contracts for those functions are as follows

`read` continues until there's an error or the message is complete

`read_some` continues until it gets at least one byte, an error
occurs, or the message is complete

In the example I posted the behaviors are very similar since the
buffer has a 512-byte limit. Either would work.

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

[Bjorn Reese via Boost]

Post by Vinnie Falco via Boost
To my understanding, chunk-extensions are a rare niche use-case with
meaning only to applications using a custom interpretation at each end
of the connection. In fact 5 years ago the IETF almost deprecated
<https://trac.ietf.org/trac/httpbis/ticket/343>

That was scary reading, because HTTPbis seemed to be unaware that such
deprecation would break some of IETF's own standards, e.g. RFC 3507.

Post by Vinnie Falco via Boost
On the other hand, I do not have significant expertise with HTTP
servers; if a compelling use-case presents itself this is an aspect of
the library which may be improved, in a backward-compatible way.

Chunk extensions were originally designed for per-chunk signatures.
I do not know how extensively this is used.

Another use-case that is used in practice is for in-band meta-data.
Consider an Internet radio station that sends a constant stream of
audio. When a new track is played this will be signaled by meta-data
telling the track title, artist name, etc. Some audio codec formats
embed this meta-data into the stream itself (e.g. MP3 ID3 tags), while
others do not. In the latter case, chunk extensions are used to carry
the meta-data.

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

[Bjorn Reese via Boost]

The chunk decorator uses boost::asio::null_buffers to overload the
field trailer.

Given your desire to adapt Beast to Networking TS, do notice that
null_buffers are not part of N4656 (they have been replaced by
dedicated wait functions.)

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost