[boost] proposal - modularize Boost build system

[Stefan Seefeld via Boost]

One of Boost's bottlenecks is its monolithic build infrastructure. It
would be beneficial for many different use cases (testing, packaging,
etc.) to decouple the building of Boost libraries (well, those that
require building, see the listing in
http://www.boost.org/doc/libs/1_64_0/more/getting_started/unix-variants.html#header-only-libraries)
such that they may optionally be built stand-alone.

At present, building a Boost library requires the entire
(super-)repository to be checked out, and the build logic itself
involves traversing the entire source tree, looking for a "Jamroot".

What would be beneficial to many would be a workflow like this:

1) Have a development environment with (some version of) Boost
pre-installed (or at least the parts that the library being built
depends on).

2) Check out a single Boost repository (e.g.,
https://github.com/boostorg/python)

3) Invoke a command to build it (if there is anything to build)

4) Invoke a command to test it

5) Invoke a command to install it

6) Invoke a command to package it (optional)


While it's of course already possible to do all the above by adding
support for another build infrastructure (point in case: Boost.Python
right now uses SCons for this very reason), this means duplication of
effort, as Boost as a whole is still built, tested (and even packaged
for binary packages) using Boost.Build, meaning I need to maintain two
sets of build infrastructure.

This proposal thus has two parts:

1) Make the nested build logic independent of the outer bits, so
individual libraries can be built stand-alone (for example, using b2 by
adding a `--with-boost` set of options to point to the location of the
prerequisite parts of Boost).

2) Define a clear interface the outer build logic will use to invoke the
nested build commands.


Note that for my own case above (Boost.Python), providing 2) would be
enough, i.e. by having a way for me to "plug in" my SCons commands to
build and test Boost.Python could obsolete the existing Boost.Build
logic as defined in Boost.Python (as well as the prerequisite parts in
Boost.Build's "python" module).

However, as my proposal is *not* actually advocating to move away from
Boost.Build, but rather to modularize it, I think 1) is essential to let
everyone else (who may not be inclined to use anything other than
Boost.Build) to also take advantage of modularization.


While I'd rather avoid delving into the technical details of how this
could possibly be implemented (if you really must, *please* do so in a
new thread !), let me outline a few use-cases that would be made
possible by the above:


* individual projects would be free to switch to their preferred
infrastructure, including CMake, SCons, etc., if they so wish

* individual projects could be much more easily developed and contributed to

* individual projects could be much more easily tested, notably in CI
environments

* individual projects could be much more easily packaged


All of the above advantages are *huge*, and reflect real-world needs,
and the technical issues to solve these are all minor. The question
really is whether there is enough will to move into that direction.

I'd be very happy to participate in the work needed to implement this.
But first we need to agree that this is where we want to go.

Thanks,
Stefan

[Peter Dimov via Boost]

Post by Stefan Seefeld via Boost
At present, building a Boost library requires the entire
(super-)repository to be checked out, and the build logic itself involves
traversing the entire source tree, looking for a "Jamroot".
1) Have a development environment with (some version of) Boost
pre-installed (or at least the parts that the library being built depends
on).
2) Check out a single Boost repository (e.g.,
https://github.com/boostorg/python)
3) Invoke a command to build it (if there is anything to build)
4) Invoke a command to test it

It's actually possible to do this with Boost.Build, but I want to talk about
something else here.

You keep wishing for that, and you keep missing the point. This is
antithetical to the original Boost spirit.

The idea of Boost is that you test your library not against the last Boost
release, or against whatever old Boost release happens to be installed. The
idea is that you test against the current development state, which today
means the develop branch.

Yes, this makes things less convenient for you, because it means that
people's changes break your build. This is on purpose. It is how Boost has
achieved its track record of stability and quality.

This is part of the price you pay for being accepted as part of Boost - the
duty to act as an integration test for your Boost dependencies. This is
beneficial for you in the long term, because you can detect breaking changes
in your dependencies before they get shipped. If you only test against 1.56,
and 1.64 breaks your library, you won't hear about it before 1.72. This does
you no good, and it does your users no good. You WANT to know if changes in
1.65's SmartPtr would break you BEFORE 1.65 gets released.

This is not theoretical. At one point in the past, certain changes in
enable_shared_from_this would have broken Boost.Python. Without it being
there to catch this fact, they would have went into a release, because
nobody else was affected.

TL;DR Boost is tested as a unit, which ensures higher quality. This is
deliberate. It's not a bad habit that needs to be broken.


_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

[Robert Ramey via Boost]

Post by Peter Dimov via Boost

Post by Stefan Seefeld via Boost
At present, building a Boost library requires the entire
(super-)repository to be checked out, and the build logic itself
involves traversing the entire source tree, looking for a "Jamroot".
1) Have a development environment with (some version of) Boost
pre-installed (or at least the parts that the library being built
depends on).
2) Check out a single Boost repository (e.g.,
https://github.com/boostorg/python)
3) Invoke a command to build it (if there is anything to build)
4) Invoke a command to test it

It's actually possible to do this with Boost.Build, but I want to talk
about something else here.
You keep wishing for that, and you keep missing the point. This is
antithetical to the original Boost spirit.

Wow - I don't get this at all.

Post by Peter Dimov via Boost
The idea of Boost is that you test your library not against the last
Boost release, or against whatever old Boost release happens to be
installed. The idea is that you test against the current development
state, which today means the develop branch.

I've think this is a very bad idea and creates lot's of problems. I've
advocated for years that we should not do this. I've pulled back on
this for the last few years because:
a) I can't convince anyone else I'm right
b) Since git let's me easily select master branch for the boost
libraries I'm not working on I can easily and unobtrusively test against
the latest release. So the problem doesn't concern me personally any more.

Post by Peter Dimov via Boost
Yes, this makes things less convenient for you, because it means that
people's changes break your build. This is on purpose. It is how Boost
has achieved its track record of stability and quality.

LOL - not because of that but in spite of that.

And what are users expected to do? One thing that gets a little lost in
this discussion is the distinction between boost library developers and
users of boost libraries in their current applications. I think some of
the CMake motivation is to help our users.

Post by Peter Dimov via Boost
This is part of the price you pay for being accepted as part of Boost -
the duty to act as an integration test for your Boost dependencies. This
is beneficial for you in the long term, because you can detect breaking
changes in your dependencies before they get shipped. If you only test
against 1.56, and 1.64 breaks your library, you won't hear about it
before 1.72. This does you no good, and it does your users no good. You
WANT to know if changes in 1.65's SmartPtr would break you BEFORE 1.65
gets released.

yada, yada, yada.

Testing is an experiment. If you change all the variables at once it's
an uncontrolled experiment and the results are useless and often
misleading. Change one variable at a time - your code.

Post by Peter Dimov via Boost
This is not theoretical. At one point in the past, certain changes in
enable_shared_from_this would have broken Boost.Python. Without it being
there to catch this fact, they would have went into a release, because
nobody else was affected.

Depending on source code from one library build to be the test of
another library is not really a test.

Post by Peter Dimov via Boost
TL;DR Boost is tested as a unit, which ensures higher quality. This is
deliberate. It's not a bad habit that needs to be broken.

This means were testing against implementation rather than
specifications. It lowers the standard of quality by permitting the
argument "well it passes the test" to be an argument. It is actually a
dis-insenetive for developers to write more tests. The more tests you
write the more you get bogged down in other persons poor code. It's part
of the motivating for those who want to diminich library dependencies
even to the point in importing source code from other libraries into our
own distribution.

But the fundamental problem is that it doesn't scale. Boost cannot grow
if the whole damn thing has to be released as a unit.


Robert Ramey


_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

[Stefan Seefeld via Boost]

Post by Peter Dimov via Boost

Post by Stefan Seefeld via Boost
At present, building a Boost library requires the entire
(super-)repository to be checked out, and the build logic itself
involves traversing the entire source tree, looking for a "Jamroot".
1) Have a development environment with (some version of) Boost
pre-installed (or at least the parts that the library being built
depends on).
2) Check out a single Boost repository (e.g.,
https://github.com/boostorg/python)
3) Invoke a command to build it (if there is anything to build)
4) Invoke a command to test it

It's actually possible to do this with Boost.Build, but I want to talk
about something else here.

Can you elaborate ? "Possible" in the sense of "it just a matter of some
programming" ? Or am I missing something ? The last times I asked
(specifically for the Boost.Python project) I was told some of the
required functionality was being worked on (on some Boost.Build
development branch), but wasn't actually neither on "master" nor "develop".

Post by Peter Dimov via Boost
You keep wishing for that, and you keep missing the point. This is
antithetical to the original Boost spirit.

Sorry, not sure what "the point" is in your reply. And not sure why you
mention the "original Boost spirit". I'm not asking what it takes to
conform to some spirit or other, I'm asking whether a specific use-case
that is obviously important to me (and to many others I gather, even
though few people express it the way I do) could be supported. Are you
telling me that it can't, because it's not in line with some "original
Boost spirit" ??

Post by Peter Dimov via Boost
The idea of Boost is that you test your library not against the last
Boost release, or against whatever old Boost release happens to be
installed. The idea is that you test against the current development
state, which today means the develop branch.

While I think that this is a problem in its own (which we could argue
about in a separate thread), let me clarify: I'm not suggesting that the
current workflow should be abandoned. I'm asking for another workflow to
be supported. I think both can coexist if that is useful.

Post by Peter Dimov via Boost
Yes, this makes things less convenient for you, because it means that
people's changes break your build. This is on purpose. It is how Boost
has achieved its track record of stability and quality.

I'm not worried about stability (as far as Boost.Python's prerequisites
are concerned, at least). I'm worried about scalability. It just doesn't
work. Or it works badly.

Post by Peter Dimov via Boost
This is part of the price you pay for being accepted as part of Boost
- the duty to act as an integration test for your Boost dependencies.
This is beneficial for you in the long term, because you can detect
breaking changes in your dependencies before they get shipped. If you
only test against 1.56, and 1.64 breaks your library, you won't hear
about it before 1.72. This does you no good, and it does your users no
good. You WANT to know if changes in 1.65's SmartPtr would break you
BEFORE 1.65 gets released.

I'v replicated this paragraph and answered in a separate thread, to keep
this discussion on-topic.

Post by Peter Dimov via Boost
This is not theoretical. At one point in the past, certain changes in
enable_shared_from_this would have broken Boost.Python. Without it
being there to catch this fact, they would have went into a release,
because nobody else was affected.

I understand, and appreciate the role downstream projects have as
"integration tests" for Boost libraries.
Again, I'm not saying this isn't useful to do. I'm asking for support
for a separate workflow.

Post by Peter Dimov via Boost
TL;DR Boost is tested as a unit, which ensures higher quality. This is
deliberate. It's not a bad habit that needs to be broken.

I understand and disagree. "Boost as a unit" just doesn't work any
longer. I think it's possible to support the kind of integration testing
you have in mind, while at the same time break up Boost into more
autonomous components.

Stefan

[Peter Dimov via Boost]

I'm asking whether a specific use-case that is obviously important to me
(and to many others I gather, even though few people express it the way I
do) could be supported.

What I use is this:

https://github.com/boostorg/system/blob/develop/.travis.yml#L288

This is not quite what you want, because it doesn't use the system-installed
Boost, it git clones one. But it only clones the minimum subset necessary
for the tests (of, in this case, Boost.System) to run.

Now admittedly in Boost.System's case this is just

$ python tools/boostdep/depinst/depinst.py system
Installing module core
Installing module predef
Installing module winapi
Installing module assert
Installing module config

taking 4 seconds, and in Boost.Python's case you'll need much more than
that. But it's (a) still not the whole Boost and (b) is only dependent on
what you #include, not on Boost's overall size.

Appendix A, Boost.Python test dependencies:

C:\Projects\boost-git\boost>dist\bin\boostdep --test python
Test dependencies for python:

assert
bind
config
conversion
core
detail
foreach
function
graph
integer
iterator
lexical_cast
mpl
numeric~conversion
preprocessor
property_map
smart_ptr
static_assert
tuple
type_traits
utility

throw_exception (from conversion)
typeof (from conversion)
range (from foreach)
move (from function)
type_index (from function)
algorithm (from graph)
any (from graph)
array (from graph)
bimap (from graph)
concept_check (from graph)
disjoint_sets (from graph)
functional (from graph)
graph_parallel (from graph)
math (from graph)
multi_index (from graph)
optional (from graph)
parameter (from graph)
property_tree (from graph)
random (from graph)
regex (from graph)
serialization (from graph)
spirit (from graph)
test (from graph)
tti (from graph)
unordered (from graph)
xpressive (from graph)
function_types (from iterator)
fusion (from iterator)
container (from lexical_cast)
predef (from mpl)
mpi (from property_map)
exception (from algorithm)
lambda (from bimap)
(unknown) (from container)
intrusive (from container)
dynamic_bitset (from graph_parallel)
filesystem (from graph_parallel)
variant (from graph_parallel)
atomic (from math)
format (from property_tree)
system (from random)
io (from serialization)
endian (from spirit)
iostreams (from spirit)
locale (from spirit)
phoenix (from spirit)
pool (from spirit)
proto (from spirit)
thread (from spirit)
timer (from test)
winapi (from system)
chrono (from thread)
date_time (from thread)
ratio (from chrono)
tokenizer (from date_time)
rational (from ratio)


_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

[Peter Dimov via Boost]

Post by Peter Dimov via Boost

I'm asking whether a specific use-case that is obviously important to me
(and to many others I gather, even though few people express it the way
I do) could be supported.

https://github.com/boostorg/system/blob/develop/.travis.yml#L288
This is not quite what you want, because it doesn't use the
system-installed Boost, ...

What you want is this:

git clone --depth=1 -b develop https://github.com/boostorg/boost.git
cd boost
git submodule update --init libs/config
git submodule update --init tools/build
git submodule update --init libs/python
./bootstrap.sh
cd libs/python
../../b2 test include=include

This works for me on CentOS 7, using the glorious preinstalled Boost 1.53.
Two tests fail, exec and import_. (And on second thought, it should work
even without include=include, and it does.)

For Travis, replace the `git submodule update --init libs/python` with a
copy of the already checked out repo, as I do in the file referenced above.

Unless you hit the CI job limit, I'd still recommend the depinst-based one
though.


_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

[Robert Ramey via Boost]

Post by Stefan Seefeld via Boost
One of Boost's bottlenecks is its monolithic build infrastructure. It
would be beneficial for many different use cases (testing, packaging,
etc.) to decouple the building of Boost libraries (well, those that
require building, see the listing in
http://www.boost.org/doc/libs/1_64_0/more/getting_started/unix-variants.html#header-only-libraries)
such that they may optionally be built stand-alone.
At present, building a Boost library requires the entire
(super-)repository to be checked out, and the build logic itself
involves traversing the entire source tree, looking for a "Jamroot".
1) Have a development environment with (some version of) Boost
pre-installed (or at least the parts that the library being built
depends on).

sort of a Boost without the libraries - a Boost developers kit?

Post by Stefan Seefeld via Boost
2) Check out a single Boost repository (e.g.,
https://github.com/boostorg/python)
3) Invoke a command to build it (if there is anything to build)
4) Invoke a command to test it
5) Invoke a command to install it

Hmmm - I don't know what this would actually do than just copy the built
library to some "special place"

Post by Stefan Seefeld via Boost
6) Invoke a command to package it (optional)

After all this time I have no idea what "packaging" means in this
context. I never could figure out what CPack is for or what it would do
for me.

Post by Stefan Seefeld via Boost
While it's of course already possible to do all the above by adding
support for another build infrastructure (point in case: Boost.Python
right now uses SCons for this very reason), this means duplication of
effort, as Boost as a whole is still built, tested (and even packaged
for binary packages) using Boost.Build, meaning I need to maintain two
sets of build infrastructure.
1) Make the nested build logic independent of the outer bits, so
individual libraries can be built stand-alone (for example, using b2 by
adding a `--with-boost` set of options to point to the location of the
prerequisite parts of Boost).

Hmmm - too fuzzy. For me, I just move to the library directory test and
invoke b2 and it builds and runs the tests. Then I just copy the built
library from the bin.v2 tree to whereever I want so I'm done. I'm
thinking this is addressed in the current system.

Post by Stefan Seefeld via Boost
2) Define a clear interface the outer build logic will use to invoke the
nested build commands.

too fuzzy. Needs to be more specific.

Post by Stefan Seefeld via Boost
Note that for my own case above (Boost.Python), providing 2) would be
enough, i.e. by having a way for me to "plug in" my SCons commands to
build and test Boost.Python could obsolete the existing Boost.Build
logic as defined in Boost.Python (as well as the prerequisite parts in
Boost.Build's "python" module).

In practice I invoke b2 with my own shell script which builds/runs the
tests and generates my own cool test matrix - library_status. Doesn't
that mean that a library/build/test can't be done from just about any
system including windows IDE, CMake etc.

Post by Stefan Seefeld via Boost
However, as my proposal is *not* actually advocating to move away from
Boost.Build, but rather to modularize it, I think 1) is essential to let
everyone else (who may not be inclined to use anything other than
Boost.Build) to also take advantage of modularization.
While I'd rather avoid delving into the technical details of how this
could possibly be implemented (if you really must, *please* do so in a
new thread !), let me outline a few use-cases that would be made
* individual projects would be free to switch to their preferred
infrastructure, including CMake, SCons, etc., if they so wish
* individual projects could be much more easily developed and contributed to
* individual projects could be much more easily tested, notably in CI
environments
* individual projects could be much more easily packaged
All of the above advantages are *huge*, and reflect real-world needs,
and the technical issues to solve these are all minor. The question
really is whether there is enough will to move into that direction.
I'd be very happy to participate in the work needed to implement this.

LOL - you'll regret this.

Post by Stefan Seefeld via Boost
But first we need to agree that this is where we want to go.

LOL - way too ambitious. How about suggestion some incremental
enhancements which would support decoupling. Example

Systems which want to support CMake build should have a CMakeList.txt
script in the library root. Systems which support Bjam Build should
have a Jamfile script in the library root, etc. There is no reason that
a library cannot support more than one build system if the library
maintainer is willing to do the work.

Bjam in the boost root would invoke bjam on on the libraries which
support bjam. CMakeList.txt in the boost root would invoke build/test
for all libraries with CMakeList.txt in the root. Bjam/CMake would
inhabit parallel universes - perhaps both simultaneously. So boost would
be in effect a practical application of ideas developed in the area of
quantum mechanics. For now, Bjam would be required - others would be
optional. In the future, this requirement might be relaxed.

FWIW I actually need both Bjam and CMake. CMake can only do a half
assed test/build of the serialization library. BUT only CMake can build
my Xcode IDE project. An xcode project has at least 1000 settings.

All in all I like this proposal and approach. It will let CMake
enthusiasts do their thing to create something that is useful to me and
users without making my life miserable. I'm actually skeptical that they
can actually do this, but this doesn't prevent me from rooting for them.

Robert Ramey



_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

[Stefan Seefeld via Boost]

Post by Robert Ramey via Boost

Post by Stefan Seefeld via Boost
One of Boost's bottlenecks is its monolithic build infrastructure. It
would be beneficial for many different use cases (testing, packaging,
etc.) to decouple the building of Boost libraries (well, those that
require building, see the listing in
http://www.boost.org/doc/libs/1_64_0/more/getting_started/unix-variants.html#header-only-libraries)
such that they may optionally be built stand-alone.
At present, building a Boost library requires the entire
(super-)repository to be checked out, and the build logic itself
involves traversing the entire source tree, looking for a "Jamroot".
1) Have a development environment with (some version of) Boost
pre-installed (or at least the parts that the library being built
depends on).

sort of a Boost without the libraries - a Boost developers kit?

Call it what you want. I thought traditionally it has been named "Boost
core". (But let's not have a bikeshed discussion about it ;-) )

Post by Robert Ramey via Boost

Post by Stefan Seefeld via Boost
2) Check out a single Boost repository (e.g.,
https://github.com/boostorg/python)
3) Invoke a command to build it (if there is anything to build)
4) Invoke a command to test it
5) Invoke a command to install it

Hmmm - I don't know what this would actually do than just copy the
built library to some "special place"

...as well as headers, documentation, etc. This is all part of the
normal Unix development culture.

Post by Robert Ramey via Boost

Post by Stefan Seefeld via Boost
6) Invoke a command to package it (optional)

After all this time I have no idea what "packaging" means in this
context. I never could figure out what CPack is for or what it would
do for me.

Likewise. There are many different packaging formats in use, depending
on the OS (e.g., "Linux distribution"), and someone needs to build those
packages. It would be helpful to provide some infrastructure for that,
as otherwise (i.e., in the current state), different OSes break up Boost
in different and incompatible ways, making it even harder for developers
using Boost to write portable code.

Post by Robert Ramey via Boost

Post by Stefan Seefeld via Boost
While it's of course already possible to do all the above by adding
support for another build infrastructure (point in case: Boost.Python
right now uses SCons for this very reason), this means duplication of
effort, as Boost as a whole is still built, tested (and even packaged
for binary packages) using Boost.Build, meaning I need to maintain two
sets of build infrastructure.
1) Make the nested build logic independent of the outer bits, so
individual libraries can be built stand-alone (for example, using b2 by
adding a `--with-boost` set of options to point to the location of the
prerequisite parts of Boost).

Hmmm - too fuzzy. For me, I just move to the library directory test
and invoke b2 and it builds and runs the tests.

By "the directory" you are surely referring to the library directory
within the superproject repo, right ? Here is again what I wrote as my

Post by Robert Ramey via Boost
2) Check out a single Boost repository (e.g.,
https://github.com/boostorg/python)
3) Invoke a command to build it (if there is anything to build)

which right now does not work.

Post by Robert Ramey via Boost
Then I just copy the built library from the bin.v2 tree to whereever
I want so I'm done. I'm thinking this is addressed in the current
system.

See above.

Post by Robert Ramey via Boost

Post by Stefan Seefeld via Boost
2) Define a clear interface the outer build logic will use to invoke the
nested build commands.

too fuzzy. Needs to be more specific.

It may appear fuzzy to you because of the above. Once you see what I
mean in 1), 2) should become clear.

Stefan

[Raffi Enficiaud via Boost]

Post by Stefan Seefeld via Boost
One of Boost's bottlenecks is its monolithic build infrastructure. It
would be beneficial for many different use cases (testing, packaging,
etc.) to decouple the building of Boost libraries (well, those that
require building, see the listing in
http://www.boost.org/doc/libs/1_64_0/more/getting_started/unix-variants.html#header-only-libraries)
such that they may optionally be built stand-alone.
At present, building a Boost library requires the entire
(super-)repository to be checked out, and the build logic itself
involves traversing the entire source tree, looking for a "Jamroot".
1) Have a development environment with (some version of) Boost
pre-installed (or at least the parts that the library being built
depends on).

If you want to easily distribute boost libraries independently, having
boost preinstalled (or already cloned or already downloaded from a
release) is not exactly what I would like to see as a user.

So my user story would be:
* I have a compiler, and the prerequisite for building boost.whatever,
which are stated by the boost.whatever readme.txt file, and that should
be light enough, or at least the minimal thing for building stuff for
the platform I am running.
* I git clone
* I build, and possibly run the tests to feel confident.

From my user perspective, all the rest is "implementation details".
This is the type of build/run I can see for other popular libraries, I
wish we get there at some point with boost.

As a library maintainer, I can say that boost.test is having hard time
to compete with other unit testing libraries such as Catch or google
test, not because of the number of files in boost.test itself, because
you need to pull a full boost release to run it, which is another level
of magnitude.

Packaging is out of the user story to my opinion, and there are nice
people very good at this. Also if installation is easy, packaging is
less important I would say (IMO).
Installation depends and can be complicated: do we want this
installation to live with another existing one (like the superproject,
or the one shipped with a eg. Debian package) ? Do we want to support
installations in user folders? etc etc When I look at all the options
that PIP INSTALL provides me with, I am sometime lost.

Now if I take my developer hat, it seems to me that there is a different
approach to take depending on the environment the library is built from:

1. I build from within boost super project
2. I build by considering boost.whatever as an entry point, and possibly
point to an existing boost installation for dependencies
3. I build on a naked system where only the build tools are installed.

What I describe above is the very simplest use case, and even with that
this is not clear to me the approach to take.

Also, I would like to point out the annoyance for users that have to
deal with the possibly many tools I have to install. You are mentioning
SCons, and I recall myself having looked at this in early 2004 and never
looked at it since. As a user, I do not want to deal with this, that is
very specific to one library. If I need to install yet-another-tool just
to satisfy some library, then sometimes this is just a no-go (sometimes
not, I am currently looking at Bazel ... but I am kind of forced to to
build TensorFlow).

The CMake thread mentioned started mentioning this, and CMake is
gathering some consensus because also, it is defacto already installed
on many of the systems developer use today. So I see Cmake not only as a
tool for building, testing etc, but also as the extra tool of less
annoyance.


There is also something a bit awkward I wanted to indicate, but we just
forgot to mention one of the most important tools so far: the **compiler**.

We cannot achieve any modularity if there is no agreement on what should
be supported, and this definitely is the most important variable to take
into account in the dependency graph of the boost.whatever libraries. If
I consider only boost.test, a lot of dependencies can just be removed
just by considering C++11, which in turn would lower dramatically the
user overhead in order to use the library.

Raffi

Post by Stefan Seefeld via Boost
2) Check out a single Boost repository (e.g.,
https://github.com/boostorg/python)
3) Invoke a command to build it (if there is anything to build)
4) Invoke a command to test it
5) Invoke a command to install it
6) Invoke a command to package it (optional)
While it's of course already possible to do all the above by adding
support for another build infrastructure (point in case: Boost.Python
right now uses SCons for this very reason), this means duplication of
effort, as Boost as a whole is still built, tested (and even packaged
for binary packages) using Boost.Build, meaning I need to maintain two
sets of build infrastructure.
1) Make the nested build logic independent of the outer bits, so
individual libraries can be built stand-alone (for example, using b2 by
adding a `--with-boost` set of options to point to the location of the
prerequisite parts of Boost).
2) Define a clear interface the outer build logic will use to invoke the
nested build commands.
Note that for my own case above (Boost.Python), providing 2) would be
enough, i.e. by having a way for me to "plug in" my SCons commands to
build and test Boost.Python could obsolete the existing Boost.Build
logic as defined in Boost.Python (as well as the prerequisite parts in
Boost.Build's "python" module).
However, as my proposal is *not* actually advocating to move away from
Boost.Build, but rather to modularize it, I think 1) is essential to let
everyone else (who may not be inclined to use anything other than
Boost.Build) to also take advantage of modularization.
While I'd rather avoid delving into the technical details of how this
could possibly be implemented (if you really must, *please* do so in a
new thread !), let me outline a few use-cases that would be made
* individual projects would be free to switch to their preferred
infrastructure, including CMake, SCons, etc., if they so wish
* individual projects could be much more easily developed and contributed to
* individual projects could be much more easily tested, notably in CI
environments
* individual projects could be much more easily packaged
All of the above advantages are *huge*, and reflect real-world needs,
and the technical issues to solve these are all minor. The question
really is whether there is enough will to move into that direction.
I'd be very happy to participate in the work needed to implement this.
But first we need to agree that this is where we want to go.
Thanks,
Stefan

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

[Stefan Seefeld via Boost]

Post by Raffi Enficiaud via Boost

Post by Stefan Seefeld via Boost
One of Boost's bottlenecks is its monolithic build infrastructure. It
would be beneficial for many different use cases (testing, packaging,
etc.) to decouple the building of Boost libraries (well, those that
require building, see the listing in
http://www.boost.org/doc/libs/1_64_0/more/getting_started/unix-variants.html#header-only-libraries)
such that they may optionally be built stand-alone.
At present, building a Boost library requires the entire
(super-)repository to be checked out, and the build logic itself
involves traversing the entire source tree, looking for a "Jamroot".
1) Have a development environment with (some version of) Boost
pre-installed (or at least the parts that the library being built
depends on).

If you want to easily distribute boost libraries independently, having
boost preinstalled (or already cloned or already downloaded from a
release) is not exactly what I would like to see as a user.

You are right. As was pointed out before, "user" stands for two distinct
roles:

* "downstream boost developer" using an "upstream boost library"
* developers of other software using any boost library

My proposal is more targeted to benefit the former. While independent
release cycles would obviously also impact the latter. But decoupling
releases is an entirely separate discussion, which I'd rather not get
into in the context of this proposal.

Post by Raffi Enficiaud via Boost
As a library maintainer, I can say that boost.test is having hard time
to compete with other unit testing libraries such as Catch or google
test, not because of the number of files in boost.test itself, because
you need to pull a full boost release to run it, which is another
level of magnitude.

Precisely, this is exactly one of the things modularization may improve.

Post by Raffi Enficiaud via Boost
Packaging is out of the user story to my opinion, and there are nice
people very good at this. Also if installation is easy, packaging is
less important I would say (IMO).

Not true in practice. Consider a developer (of the second category
above) on Linux. Different distributions use different ways to split
Boost libraries into packages, so to be portable, I need to check for
them all. That's a huge maintenance burden which could be solved if some
basic support for packaging would be built right into the Boost build
system (be it Boost.Build or something else).

Of course you could argue that in many real-world cases even that
use-case isn't very common, as lots of commercial software is built by
cloning Boost and building that in-house. I think that's very
unfortunate if unavoidable, given the current state of things. See my
other proposal (about stability / compatibility) for how to improve
things on that front.

Post by Raffi Enficiaud via Boost
There is also something a bit awkward I wanted to indicate, but we
just forgot to mention one of the most important tools so far: the
**compiler**.
We cannot achieve any modularity if there is no agreement on what
should be supported, and this definitely is the most important
variable to take into account in the dependency graph of the
boost.whatever libraries. If I consider only boost.test, a lot of
dependencies can just be removed just by considering C++11, which in
turn would lower dramatically the user overhead in order to use the
library.

True. But again, let's not kill the discussion by widening the scope
further. For the sake of the discussion let's assume that the compiler
is a pre-defined constant, rather than an additional (almost free)
parameter.

Stefan

[Raffi Enficiaud via Boost]

Post by Stefan Seefeld via Boost

Post by Raffi Enficiaud via Boost

Post by Stefan Seefeld via Boost
One of Boost's bottlenecks is its monolithic build infrastructure. It
would be beneficial for many different use cases (testing, packaging,
etc.) to decouple the building of Boost libraries (well, those that
require building, see the listing in
http://www.boost.org/doc/libs/1_64_0/more/getting_started/unix-variants.html#header-only-libraries)
such that they may optionally be built stand-alone.
At present, building a Boost library requires the entire
(super-)repository to be checked out, and the build logic itself
involves traversing the entire source tree, looking for a "Jamroot".
1) Have a development environment with (some version of) Boost
pre-installed (or at least the parts that the library being built
depends on).

If you want to easily distribute boost libraries independently, having
boost preinstalled (or already cloned or already downloaded from a
release) is not exactly what I would like to see as a user.

You are right. As was pointed out before, "user" stands for two distinct
* "downstream boost developer" using an "upstream boost library"
* developers of other software using any boost library
My proposal is more targeted to benefit the former. While independent
release cycles would obviously also impact the latter. But decoupling
releases is an entirely separate discussion, which I'd rather not get
into in the context of this proposal.

Post by Raffi Enficiaud via Boost
As a library maintainer, I can say that boost.test is having hard time
to compete with other unit testing libraries such as Catch or google
test, not because of the number of files in boost.test itself, because
you need to pull a full boost release to run it, which is another
level of magnitude.

Precisely, this is exactly one of the things modularization may improve.

Post by Raffi Enficiaud via Boost
Packaging is out of the user story to my opinion, and there are nice
people very good at this. Also if installation is easy, packaging is
less important I would say (IMO).

Not true in practice. Consider a developer (of the second category
above) on Linux. Different distributions use different ways to split
Boost libraries into packages, so to be portable, I need to check for
them all. That's a huge maintenance burden which could be solved if some
basic support for packaging would be built right into the Boost build
system (be it Boost.Build or something else).

Maybe I misunderstood, but I thought that you were suggesting that we
include the possibility to package boost or any of its component
"easily" in boost itself.

First of all, my understanding of packaging is: I can create "packages"
that indicate their dependencies and the way they can be built such that
I can install a library on my OS and remove it afterwards (like .deb,
pip install, and in some extent brew).

To me this goes to the distros/OS packager desk, like the Debian
packagers or brew people. Also for some, whatever good packaging support
you provide them with, they will just not use and use their own system
instead, because it is stable and they are comfortable with. So whatever
initiative we take, IMO, within boost concerning that is a bit useless.

I see packaging as hard, I do not want to do it, not even starting
thinking of it :) Also for users, packaging is a function taking as
first parameter the OS version, rather than the boost version. There are
things like PPA but this is not really helping the end user I have to say.

Post by Stefan Seefeld via Boost
Of course you could argue that in many real-world cases even that
use-case isn't very common, as lots of commercial software is built by
cloning Boost and building that in-house. I think that's very
unfortunate if unavoidable, given the current state of things. See my
other proposal (about stability / compatibility) for how to improve
things on that front.

Well, after almost two decades of boost, I think now (at least on linux)
ppl know that there is no ABI compatibility among minor versions of
boost, but only between patch version (and since there is barely patch
release ...). Which means that the way boost make version number is just
different from the "convention" (whatever convention means).

To sum up, I would rather limit the discussion at the "installation"
level. As a user of boost.whatever, I want to install boost.whatever
once and use it many times. I think this impacts the two notions of
users you mentioned.

Post by Stefan Seefeld via Boost

Post by Raffi Enficiaud via Boost
There is also something a bit awkward I wanted to indicate, but we
just forgot to mention one of the most important tools so far: the
**compiler**.
We cannot achieve any modularity if there is no agreement on what
should be supported, and this definitely is the most important
variable to take into account in the dependency graph of the
boost.whatever libraries. If I consider only boost.test, a lot of
dependencies can just be removed just by considering C++11, which in
turn would lower dramatically the user overhead in order to use the
library.

True. But again, let's not kill the discussion by widening the scope
further. For the sake of the discussion let's assume that the compiler
is a pre-defined constant, rather than an additional (almost free)
parameter.

Right. Let's call the pruning of the dependency graph an optimization.





_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

[Edward Diener via Boost]

Post by Stefan Seefeld via Boost
One of Boost's bottlenecks is its monolithic build infrastructure. It
would be beneficial for many different use cases (testing, packaging,
etc.) to decouple the building of Boost libraries (well, those that
require building, see the listing in
http://www.boost.org/doc/libs/1_64_0/more/getting_started/unix-variants.html#header-only-libraries)
such that they may optionally be built stand-alone.
At present, building a Boost library requires the entire
(super-)repository to be checked out, and the build logic itself
involves traversing the entire source tree, looking for a "Jamroot".
1) Have a development environment with (some version of) Boost
pre-installed (or at least the parts that the library being built
depends on).
2) Check out a single Boost repository (e.g.,
https://github.com/boostorg/python)
3) Invoke a command to build it (if there is anything to build)
4) Invoke a command to test it
5) Invoke a command to install it
6) Invoke a command to package it (optional)
While it's of course already possible to do all the above by adding
support for another build infrastructure (point in case: Boost.Python
right now uses SCons for this very reason), this means duplication of
effort, as Boost as a whole is still built, tested (and even packaged
for binary packages) using Boost.Build, meaning I need to maintain two
sets of build infrastructure.
1) Make the nested build logic independent of the outer bits, so
individual libraries can be built stand-alone (for example, using b2 by
adding a `--with-boost` set of options to point to the location of the
prerequisite parts of Boost).
2) Define a clear interface the outer build logic will use to invoke the
nested build commands.
Note that for my own case above (Boost.Python), providing 2) would be
enough, i.e. by having a way for me to "plug in" my SCons commands to
build and test Boost.Python could obsolete the existing Boost.Build
logic as defined in Boost.Python (as well as the prerequisite parts in
Boost.Build's "python" module).
However, as my proposal is *not* actually advocating to move away from
Boost.Build, but rather to modularize it, I think 1) is essential to let
everyone else (who may not be inclined to use anything other than
Boost.Build) to also take advantage of modularization.
While I'd rather avoid delving into the technical details of how this
could possibly be implemented (if you really must, *please* do so in a
new thread !), let me outline a few use-cases that would be made
* individual projects would be free to switch to their preferred
infrastructure, including CMake, SCons, etc., if they so wish
* individual projects could be much more easily developed and contributed to
* individual projects could be much more easily tested, notably in CI
environments
* individual projects could be much more easily packaged
All of the above advantages are *huge*, and reflect real-world needs,
and the technical issues to solve these are all minor. The question
really is whether there is enough will to move into that direction.
I'd be very happy to participate in the work needed to implement this.
But first we need to agree that this is where we want to go.

A serious problem to consider, whenever anyone speaks of a modularized
Boost where a library can be distributed on its own, is library
dependencies. This problem exists for all libraries, not just Boost. But
most libraries have their own dependency system, usually based on
whatever OS that library is being used, whereas Boost libraries almost
always intend to be cross-platform. How should Boost solve this problem
is for me the bottleneck of distributing a particular Boost library.

When I speak of dependencies I am not just speaking of library X
depending on other libraries A, B, and C etc. I am also speaking of
library X depending on particular versions of library A, B, C etc. But
since the only versioning system Boost has is a single version number
for a Boost release, and since their is no way a library can check even
that single version number of a Boost release either at compile or
run-time, Boost libraries have no way to check versioning of other
individual Boost libraries on which a library may depend.

If you say, I am going to distribute my library X with particular
releases of library A, B, and C etc. with which I know my library X will
work correctly you then have end-users of your library who may have
multiple copies of libraries X, A, B, and C etc. on their systems,
running each set within its own environment, not knowing how to identify
each set of libraries, and hoping they can run these things and avoid
the well-known shared library hell which has plagued end-users for
years. Let's be realistic, this is a real problem which only some sort
of Boost individual library versioning system for starters can hope to
solve.

Post by Stefan Seefeld via Boost
Thanks,
Stefan

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

[Robert Ramey via Boost]

Post by Edward Diener via Boost
Let's be realistic, this is a real problem which only some sort
of Boost individual library versioning system for starters can hope to
solve.

If I may be so bold as to summarize your point:

In order to distribute boost as individual libraries as opposed to a
monolithic set, individual library versioning will sooner or later have
to be adopted.

I think this is indisputable. But I don't think we have to worry about
in practice. Whatever we do, it will take sometime to get there and, if
we ever do get there, I think that adding this feature won't be a big
problem. Of course if we don't ever get there, we've got nothing
nothing to worry about.

Robert Ramey



_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

[Ion Gaztañaga via Boost]

Post by Robert Ramey via Boost

Let's be realistic, this is a real problem which only some sort of
Boost individual library versioning system for starters can hope to
solve.

In order to distribute boost as individual libraries as opposed to a
monolithic set, individual library versioning will sooner or later have
to be adopted.
I think this is indisputable. But I don't think we have to worry about
in practice. Whatever we do, it will take sometime to get there and, if
we ever do get there, I think that adding this feature won't be a big
problem. Of course if we don't ever get there, we've got nothing
nothing to worry about.

I respectfully disagree.

If you want to release your library individually, then it's no longer
Boost. Boost is a coherent library collection. You are free to release
your library individually, just like ASIO does. You'll need to somehow
solve dependency and compatibility issues on other boost libraries yourself.

If git clone is huge for Boost, then it's a git user problem, because
it's a decentralized VCS, just use a shallow clone.

If we further modularize Boost libraries, then someone will propose that
each library should choose its VCS, bug system and mailing list.

I don't like each library to use a different build tool (CMake, SCons,
etc...) I like the fact that I can write my test jamfile triggers the
creation of any dependent library just because all of them use bjam and
other Boost libraries are designed to act friendly with my library.

If you want to have a Boost library you need to maintain the style and
rules of Boost. If you want to be a standalone library then you can
already do that, but don't call it Boost. If we want to say those
standalone libraries are somewhat related to Boost, then let's invent
another name and define more relaxed rule for them.

Best,

Ion

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

[Robert Ramey via Boost]

Post by Ion Gaztañaga via Boost
If you want to release your library individually, then it's no longer
Boost.

Hmmm - now we're getting down to "what is boost" ....

Post by Ion Gaztañaga via Boost
Boost is a coherent library collection.

some might differ on this point
You are free to release

Post by Ion Gaztañaga via Boost
your library individually, just like ASIO does.

thank you

You'll need to somehow

Post by Ion Gaztañaga via Boost
solve dependency and compatibility issues on other boost libraries yourself.

Hmmm - honestly, boost doesn't do as much in this area as one would
think. Bjam does handle dependencies - true. Compatibity is managed
through test. I don't think any one as suggested changing any of this.
I think (though I'm not actually sure) that this discussion is to
facilitate the usage of CMake by boost users to don't want to be boost
developers but rather "just" boost users.

Post by Ion Gaztañaga via Boost
If git clone is huge for Boost, then it's a git user problem, because
it's a decentralized VCS, just use a shallow clone.
If we further modularize Boost libraries, then someone will propose that
each library should choose its VCS, bug system and mailing list.

Actually, I already proposed this some time ago. In fact we already
have much of that. For example, for bugs some libraries use git issues
while others use the traditional system. Each library chooses it's own
documentation tools. The git submodule implementation could be seen as
each library having it's own VCS just tied together at the top.

Post by Ion Gaztañaga via Boost
I don't like each library to use a different build tool (CMake, SCons,
etc...) I like the fact that I can write my test jamfile triggers the
creation of any dependent library just because all of them use bjam and

we're not talking about what you want to do as a boost developer. You
can do whatever you want. The question is should you, boost or anyone
else tell developers of other libraries what they should do?

Post by Ion Gaztañaga via Boost
other Boost libraries are designed to act friendly with my library.

Right - but only at the source code and local build level. For users
using a portion of boost in their apps, they don't see it this way.

Post by Ion Gaztañaga via Boost
If you want to have a Boost library you need to maintain the style and
rules of Boost.

Hmm - boost has a lot of rules related to the source code, directory
structure, requirements for tests, etc. I don't see this as being impacted.

Post by Ion Gaztañaga via Boost
If you want to be a standalone library then you can
already do that, but don't call it Boost. If we want to say those
standalone libraries are somewhat related to Boost, then let's invent
another name and define more relaxed rule for them.

If one of the promoters of CMake want to make a "thing" which
incorporates the most recent version of boost source code by reference,
I wouldn't object. They could call it "modular boost". But I doubt
they'll do it. It's really only an appealing idea if someone else does
the actual work.

Robert Ramey

Post by Ion Gaztañaga via Boost
Best,
Ion
_______________________________________________
http://lists.boost.org/mailman/listinfo.cgi/boost

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mai

[Ion Gaztañaga via Boost]

Post by Robert Ramey via Boost
Actually, I already proposed this some time ago. In fact we already
have much of that. For example, for bugs some libraries use git issues
while others use the traditional system. Each library chooses it's own
documentation tools. The git submodule implementation could be seen as
each library having it's own VCS just tied together at the top.

We have both the old a the new bug system, but no library uses Bugzilla
for that. That said, I would like to migrate all the old issues from
track to github. I prefer a trac-style bug management system instead of
pull requests, but unification has many advantages. Common tools help
maintenance and collaboration between boost developers and any abandoned
library can be rescued because all used tools are familiar. They help
reviews.

Documentation is a different issue, we can't compare the coupling
between libraries to the different documentation styles. In any case,
many Boost libraries use Quickbook or Boostbook which IMHO should be
encouraged.

Post by Robert Ramey via Boost

Post by Ion Gaztañaga via Boost
I don't like each library to use a different build tool (CMake, SCons,
etc...) I like the fact that I can write my test jamfile triggers the
creation of any dependent library just because all of them use bjam and

we're not talking about what you want to do as a boost developer. You
can do whatever you want. The question is should you, boost or anyone
else tell developers of other libraries what they should do?

Boost is voluntary organization. Organizations have rules that try to
help the goal of the organization. If you want to call you library
"Boost" you need to follow the rules. This makes a lot of sense to me. I
could release my library directly in Github, but following Boost rules
my library is interoperable with other libraries, and my dependencies
don't break often. It's not as flexible as I might want but it has a lot
of advantages.

If each library is free to choose the build system, release dates, track
system... what's the point of naming it "Boost"? Just because they were
reviewed in the boost mailing list? I need to build several Boost
libraries as my library depends on them. Do I need to learn different
build tools just to run my tests?

Post by Robert Ramey via Boost

Post by Ion Gaztañaga via Boost
other Boost libraries are designed to act friendly with my library.

Right - but only at the source code and local build level. For users
using a portion of boost in their apps, they don't see it this way.

Post by Ion Gaztañaga via Boost
If you want to have a Boost library you need to maintain the style and
rules of Boost.

Hmm - boost has a lot of rules related to the source code, directory
structure, requirements for tests, etc. I don't see this as being impacted.

The impact on how other libraries build, name, find dependencies, ... is
much more important to my library than how the source code of those
libraries is written.

I understand that some aspects of Boost don't work as desired/expected,
but I doubt any "modularization" that allows different build/trac
systems will solve them. IMHO the entropy can only increase.

Best,

Ion

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

[Edward Diener via Boost]

Post by Ion Gaztañaga via Boost

Post by Robert Ramey via Boost

Let's be realistic, this is a real problem which only some sort of
Boost individual library versioning system for starters can hope to
solve.

In order to distribute boost as individual libraries as opposed to a
monolithic set, individual library versioning will sooner or later
have to be adopted.
I think this is indisputable. But I don't think we have to worry
about in practice. Whatever we do, it will take sometime to get there
and, if we ever do get there, I think that adding this feature won't
be a big problem. Of course if we don't ever get there, we've got
nothing nothing to worry about.

I respectfully disagree.
If you want to release your library individually, then it's no longer
Boost. Boost is a coherent library collection. You are free to release
your library individually, just like ASIO does. You'll need to somehow
solve dependency and compatibility issues on other boost libraries yourself.

If you think about this a little more deeply I think you will understand
that this methodology will not work.

As in:

1) You release Boost library X which uses some version of Boost library A.

2) Boost library Y is released which uses a different version of Boost
library A.

3) The end-user attempts to use library X and library Y in the same TU.

This has absolutely nothing to do with you releasing library X and
"somehow solve dependency and compatibility issues on other boost
libraries yourself".

The current equivalent to this with monolithic Boost is the end-user
using Boost 1.63 and Boost 1.64 together in his own module ( library or
executable ). The end-user knows enough not to do this because for
either 1.63 or 1.64 all libraries are "guaranteed" to work in tandem.
But this can't happen with individual Boost libraries ( and their
dependencies ) being released by themselves without any form of versioning.

Post by Ion Gaztañaga via Boost
If git clone is huge for Boost, then it's a git user problem, because
it's a decentralized VCS, just use a shallow clone.
If we further modularize Boost libraries, then someone will propose that
each library should choose its VCS, bug system and mailing list.
I don't like each library to use a different build tool (CMake, SCons,
etc...) I like the fact that I can write my test jamfile triggers the
creation of any dependent library just because all of them use bjam and
other Boost libraries are designed to act friendly with my library.
If you want to have a Boost library you need to maintain the style and
rules of Boost. If you want to be a standalone library then you can
already do that, but don't call it Boost. If we want to say those
standalone libraries are somewhat related to Boost, then let's invent
another name and define more relaxed rule for them.
Best,
Ion

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listin

[Stefan Seefeld via Boost]

Post by Robert Ramey via Boost

Let's be realistic, this is a real problem which only some sort of
Boost individual library versioning system for starters can hope to
solve.

In order to distribute boost as individual libraries as opposed to a
monolithic set, individual library versioning will sooner or later
have to be adopted.
I think this is indisputable. But I don't think we have to worry
about in practice. Whatever we do, it will take sometime to get there
and, if we ever do get there, I think that adding this feature won't
be a big problem. Of course if we don't ever get there, we've got
nothing nothing to worry about.

I watch (somewhat in horror, I have to admit) the follow-up mails as
they predictably derail into un-manageable scenarios. So I'd like to
point out that my proposal in no way implies any particular release
policy, i.e. whether individual libraries are released independently or not.
So when you dive into that discussion, please be aware that it's
entirely orthogonal to the proposal at hand. Whether or not releasing
Boost libraries as independent entities has no bearing on the usefulness
or feasibility of modularising Boost.

Thank you,
Stefan

[Edward Diener via Boost]

Post by Stefan Seefeld via Boost

Post by Robert Ramey via Boost

Let's be realistic, this is a real problem which only some sort of
Boost individual library versioning system for starters can hope to
solve.

In order to distribute boost as individual libraries as opposed to a
monolithic set, individual library versioning will sooner or later
have to be adopted.
I think this is indisputable. But I don't think we have to worry
about in practice. Whatever we do, it will take sometime to get there
and, if we ever do get there, I think that adding this feature won't
be a big problem. Of course if we don't ever get there, we've got
nothing nothing to worry about.

I watch (somewhat in horror, I have to admit) the follow-up mails as
they predictably derail into un-manageable scenarios. So I'd like to
point out that my proposal in no way implies any particular release
policy, i.e. whether individual libraries are released independently or not.
So when you dive into that discussion, please be aware that it's
entirely orthogonal to the proposal at hand. Whether or not releasing
Boost libraries as independent entities has no bearing on the usefulness
or feasibility of modularising Boost.

You did mention in your OP:

5) Invoke a command to install it

6) Invoke a command to package it (optional)

Maybe you need to be more specific about what you mean in each case. It
sure sounds to me, by the 2 items above, as if you meant to suggest that
you could distribute an individual Boost library ( and its dependencies
) separately from the current monolithic Boost tree. If so, I am
suggesting that without a very well worked out versioning system for
individual Boost libraries such a plan will end up with serious problems
for the end-user.

Post by Stefan Seefeld via Boost
Thank you,
Stefan

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

[Peter Dimov via Boost]

Post by Stefan Seefeld via Boost
5) Invoke a command to install it

Ah, I forgot about those, thanks Edward.

./b2 --with-python stage

There is a subtlety here, one which might not matter in practice. This
libboost_python will be compiled against the checked out Config, instead of
the 1.53 preinstalled one. I'm pretty sure that this will not cause any
problems, but if you want to be strict, you'll have to murder -rf
libs/config/include before starting the test/stage procedure.

The reason for this is that currently a part of Boost.Build depends on a
part of Boost.Config (for the feature checks such as cxx11_smart_ptr I see
Boost.Python uses) so libs/config needs to be checked out for tools/build to
work. But the actual C++ portion of Boost.Config isn't needed, we could use
the preinstalled one.

Post by Stefan Seefeld via Boost
6) Invoke a command to package it (optional)

On your own here, sorry.


_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

[Stefan Seefeld via Boost]

Post by Stefan Seefeld via Boost
5) Invoke a command to install it
6) Invoke a command to package it (optional)
Maybe you need to be more specific about what you mean in each case.
It sure sounds to me, by the 2 items above, as if you meant to suggest
that you could distribute an individual Boost library ( and its
dependencies ) separately from the current monolithic Boost tree.

I did indeed.

Post by Stefan Seefeld via Boost
If so, I am suggesting that without a very well worked out versioning
system for individual Boost libraries such a plan will end up with
serious problems for the end-user.

Distributing boost libraries separately doesn't imply that I don't
respect Boost's releases or version numbers. Packaging and releasing are
orthogonal concepts.
See for example https://apps.fedoraproject.org/packages/boost and
https://packages.debian.org/jessie/libboost1.55-dev. Fedora and Debian
are two popular Linux distributions, and they do of course provide
separate Boost packages.

But given the lack of guidelines (or infrastructure) to package Boost
components (which includes not just libraries, but also tools to build
them and their documentation), they differ...

Stefan

[Peter Dimov via Boost]

Packaging and releasing are orthogonal concepts.

Not really. Releasing separately from Boost means that you now have two
Boost versions to communicate: the version of Boost.Python, and the version
of the preinstalled Boost against which it was built (I built, f.ex.
libboost-python-1.65.0-1.53.0 in my CentOS 7 case.)

This consequently affects your downstream dependencies, which also have to
include these two versions, in addition to their own. So when a library X
depends on Boost.Python and it's compiled against Boost.Python
1.65.0-1.53.0, its version would now be X-1.17-1.65.0-1.53.0. This is not
compatible with Y-2.1-1.63.0-1.53.0, even though both are built in the same
CentOS 7 environment.


_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

[Stefan Seefeld via Boost]

Post by Peter Dimov via Boost

Packaging and releasing are orthogonal concepts.

Not really. Releasing separately from Boost means that you now have
two Boost versions to communicate: the version of Boost.Python, and
the version of the preinstalled Boost against which it was built (I
built, f.ex. libboost-python-1.65.0-1.53.0 in my CentOS 7 case.)
This consequently affects your downstream dependencies, which also
have to include these two versions, in addition to their own. So when
a library X depends on Boost.Python and it's compiled against
Boost.Python 1.65.0-1.53.0, its version would now be
X-1.17-1.65.0-1.53.0. This is not compatible with Y-2.1-1.63.0-1.53.0,
even though both are built in the same CentOS 7 environment.

I have no idea why you are mentioning all this. How often do I need to
repeat that this proposal is about modularization, not about decoupling
the release process of individual Boost components ?

Stefan

[Peter Dimov via Boost]

Post by Stefan Seefeld via Boost
I have no idea why you are mentioning all this. How often do I need to
repeat that this proposal is about modularization, not about decoupling
the release process of individual Boost components ?

All right. What, specifically, do you want further modularized? Did you even
read what I wrote in my other posts?


_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

[Stefan Seefeld via Boost]

Post by Peter Dimov via Boost

Post by Stefan Seefeld via Boost
I have no idea why you are mentioning all this. How often do I need
to repeat that this proposal is about modularization, not about
decoupling the release process of individual Boost components ?

All right. What, specifically, do you want further modularized? Did
you even read what I wrote in my other posts?

I did, yes. Sorry for not specifically acknowledging that. I know about
these tricks (I use something similar to set up my Boost.Python CI
environment), but consider them hacks as they don't really solve the
fundamental requests, which are:

* I want to be able to build a given Boost library stand-alone, with
nothing but the library's repo being checked out. (In other words:
prerequisite Boost components should be assumed pre-installed.)
* I want to be able to choose the build (, test, etc.) infrastructure
used for my library, and make sure that when Boost is built as a whole,
that build infrastructure is used.

I can't stress this enough: this second point is to 99% non-technical.
It's about letting project maintainers decide how they develop (build,
test, etc.) so we don't have to have these "lets all switch from tool A
to tool B" discussions any longer. The choice between b2, cmake, scons
should be done per project, rather than the whole Boost organization at
once. The only technical aspect of this is the definition of the
interface between super-project and sub-project, i.e. the mechanism by
which Boost.Build invokes the library-specific build systems for those
of us who want to continue building Boost as a whole.

Stefan

[Peter Dimov via Boost]

I know about these tricks (I use something similar to set up my
Boost.Python CI environment), but consider them hacks as they don't really
* I want to be able to build a given Boost library stand-alone, with
prerequisite Boost components should be assumed pre-installed.)

Why is this so important for you? The difference is just one superproject
shell and one tools/build. Is this a matter of principle, or are there
technical reasons?

Anyway.

Boost.Build actually supports this use case:

sudo yum install boost-jam
sudo yum install boost-build
git clone --depth=1 https://github.com/boostorg/python
cd python
touch Jamroot
bjam test

The first error here is

IMPORT error: rule "requires" unknown in module
"../../config/checks/config"

because we don't have Boost.Config checked out as a sibling here.

When I comment out your use of ../../config/checks/config in test/Jamfile,
it errors out with

rule numpy_test unknown

Trying to build with

bjam build

fails with

rule py-version unknown

Both of these errors are because these rules don't exist in the 1.53
python.jam.

So frankly, I'm not sure how do you suggest this needs to be addressed. On
one hand, you want to use the system Boost.Build, and on the other, your
Jamfiles depend on features that aren't present in it. This simply cannot
work, your desires are contradictory.


_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

[Stefan Seefeld via Boost]

Post by Peter Dimov via Boost

I know about these tricks (I use something similar to set up my
Boost.Python CI environment), but consider them hacks as they don't
* I want to be able to build a given Boost library stand-alone, with
prerequisite Boost components should be assumed pre-installed.)

Why is this so important for you? The difference is just one
superproject shell and one tools/build. Is this a matter of principle,
or are there technical reasons?

Both. I want to control the environment in which my library is being
built (, tested, etc.), and thus I don't want to fetch that environment
from a repository and build it on-the-fly.

Post by Peter Dimov via Boost
Anyway.
sudo yum install boost-jam
sudo yum install boost-build
git clone --depth=1 https://github.com/boostorg/python
cd python
touch Jamroot
bjam test
The first error here is
IMPORT error: rule "requires" unknown in module
"../../config/checks/config"
because we don't have Boost.Config checked out as a sibling here.
When I comment out your use of ../../config/checks/config in
test/Jamfile, it errors out with
rule numpy_test unknown
Trying to build with
bjam build
fails with
rule py-version unknown
Both of these errors are because these rules don't exist in the 1.53
python.jam.
So frankly, I'm not sure how do you suggest this needs to be addressed.

The numpy_test rule should probably have been added to the local Jamfile
(i.e., be part of Boost.Python, rather than Boost.Build). I'm not sure
about the config checks. Arguably they are part of the build system, and
thus should be included in Boost.Build, rather than a separate Boost
library.

Post by Peter Dimov via Boost
On one hand, you want to use the system Boost.Build, and on the other,
your Jamfiles depend on features that aren't present in it. This
simply cannot work, your desires are contradictory.

Not sure what your point is. I think it's perfectly normal that on some
platforms the default (system) version of a package is too old, so a
newer version needs to be pulled in from another repo ("testing",
perhaps ?). That doesn't invalidate my desire to work with system
packages though, rather than development versions.

Stefan

[Peter Dimov via Boost]

Post by Stefan Seefeld via Boost

Post by Peter Dimov via Boost
On one hand, you want to use the system Boost.Build, and on the other,
your Jamfiles depend on features that aren't present in it. This simply
cannot work, your desires are contradictory.

Not sure what your point is. I think it's perfectly normal that on some
platforms the default (system) version of a package is too old, so a newer
version needs to be pulled in from another repo ("testing", perhaps ?).
That doesn't invalidate my desire to work with system packages though,
rather than development versions.

My point is that the reason it doesn't work is that your Jamfiles are using
features not present in the system Boost.Build.

You were complaining that we need to make it work. But there's no deficiency
to be addressed here. It already works.

So what's the complaint?


_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

[Stefan Seefeld via Boost]

Post by Peter Dimov via Boost

Post by Peter Dimov via Boost
On one hand, you want to use the system Boost.Build, and on the

other, > your Jamfiles depend on features that aren't present in it.
This simply > cannot work, your desires are contradictory.
Not sure what your point is. I think it's perfectly normal that on
some platforms the default (system) version of a package is too old,
so a newer version needs to be pulled in from another repo
("testing", perhaps ?). That doesn't invalidate my desire to work
with system packages though, rather than development versions.

My point is that the reason it doesn't work is that your Jamfiles are
using features not present in the system Boost.Build.
You were complaining that we need to make it work. But there's no
deficiency to be addressed here. It already works.
So what's the complaint?

I'm not "complaining", I'm *proposing* an architectural change. Again:

As a boost library maintainer I want to decouple my library from the
rest of Boost to be able to

* build it as a unit (i.e., with everything else being fixed as a
prerequisite, rather than being built on-the-fly)
* use tools of my own choice to build, test, package, issue-track,
document (etc., etc.) my library, so we won't need to agree on this
scale whether to use tool A or tool B. (Recall the question triggering
this proposal was the proposal *for the entirely of Boost libraries* to
switch from Boost.Build to CMake.)

I'd be more than happy to learn that this is already possible, at which
point I'd write an article (a wiki page, say) to document how to do it.
So let's assume that creating a "Jamroot" file in my library's root
directory is all it takes to let b2 build my library stand-alone.

Post by Peter Dimov via Boost
2) Define a clear interface the outer build logic will use to invoke the
nested build commands.

In other words, what does that Jamroot file need to contain at a
minimum, to satisfy the global build processes (i.e., the ones used to
build Boost as a whole, including building release docs etc.) ? There
are globally called rules such as "boost-install", "boostrelease", etc.
that seem to be required. And what about parameters such as build
variants or toolchain versions ? How can I intercept those such that I
can call my own (local) build logic ? Is that documented anywhere ?

Stefan

[Peter Dimov via Boost]

Post by Stefan Seefeld via Boost
2) Define a clear interface the outer build logic will use to invoke the
nested build commands.

The clear interface at present is that you need to have a Jamfile. Building
using something else as part of the global build is not supported, unless
you somehow invoke this something else from your Jamfile. But this would
imply that whatever something else you pick - for instance, SCons - would
now become a prerequisite for building Boost.

In other words, what does that Jamroot file need to contain at a minimum,
to satisfy the global build processes (i.e., the ones used to build Boost
as a whole, including building release docs etc.) ?

The global build process currently in use requires you to not have a
Jamroot.

There are globally called rules such as "boost-install", "boostrelease",
etc. that seem to be required.

Yes, if you want to use the boost-install rule, it won't work without the
global Boost Jamroot.

I'm not entirely clear on what we're talking about here though. Are we in
the "git clone boostorg/python" standalone case yet, or are we in "git
clone --recursive boostorg/boost", except you want to use SCons for
libs/python instead?


_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

[Stefan Seefeld via Boost]

Post by Peter Dimov via Boost

Post by Stefan Seefeld via Boost
2) Define a clear interface the outer build logic will use to

invoke the > nested build commands.

The clear interface at present is that you need to have a Jamfile.
Building using something else as part of the global build is not
supported, unless you somehow invoke this something else from your
Jamfile. But this would imply that whatever something else you pick -
for instance, SCons - would now become a prerequisite for building Boost.

In other words, what does that Jamroot file need to contain at a
minimum, to satisfy the global build processes (i.e., the ones used
to build Boost as a whole, including building release docs etc.) ?

The global build process currently in use requires you to not have a
Jamroot.

There are globally called rules such as "boost-install",
"boostrelease", etc. that seem to be required.

Yes, if you want to use the boost-install rule, it won't work without
the global Boost Jamroot.
I'm not entirely clear on what we're talking about here though. Are we
in the "git clone boostorg/python" standalone case yet, or are we in
"git clone --recursive boostorg/boost", except you want to use SCons
for libs/python instead?

*Sigh*. Am I expressing myself really that poorly ?

I want more autonomy / independence for individual libraries. I want to
be able to build them stand-alone, which you tell me already works if I
have toplevel Jamroot in my repo, except you then tell me that I may in
fact not because the global build process requires me not to have that.

I also want to be able to pick my own build (etc.) tools, not in
addition to Boost.Build, but instead of it. I understand that right now
that's not supported, which is why I'm writing this proposal. What would
it take for Boost to support individual libraries to be built with
anything else ? What requirements would that "anything" have to meet,
and how would it interact with the existing infrastructure to work ? Is
that such a strange request ?

Stefan

[Peter Dimov via Boost]

What would it take for Boost to support individual libraries to be built
with anything else ?

In what scenario? Standalone, or as part of the Boost release?

If standalone, it's up to you to support whatever you like.

If as part of the release, this would mean that everyone who wants to build
a Boost release would now need to have your preferred build system
installed. Currently, we don't require anything else, as Boost.Build is part
of the release. So this would be a significant regression in usability.


_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

[Stefan Seefeld via Boost]

Post by Peter Dimov via Boost

What would it take for Boost to support individual libraries to be
built with anything else ?

In what scenario? Standalone, or as part of the Boost release?

Both, as the goal is not to add more infrastructure, but to replace it.

Post by Peter Dimov via Boost
If standalone, it's up to you to support whatever you like.
If as part of the release, this would mean that everyone who wants to
build a Boost release would now need to have your preferred build
system installed. Currently, we don't require anything else, as
Boost.Build is part of the release. So this would be a significant
regression in usability.

I understand. This is a bit of a vicious circle: Right now Boost is
always built as a whole, so lots of people do it. In a modular Boost
world, fewer people would build all of boost, as it's much easier to
build just the libraries people need.

Stefan

[Tom Kent via Boost]

On Mon, Jun 19, 2017 at 2:57 PM, Stefan Seefeld via Boost <

Post by Stefan Seefeld via Boost

Post by Peter Dimov via Boost

What would it take for Boost to support individual libraries to be
built with anything else ?

In what scenario? Standalone, or as part of the Boost release?

Both, as the goal is not to add more infrastructure, but to replace it.

Post by Peter Dimov via Boost
If standalone, it's up to you to support whatever you like.
If as part of the release, this would mean that everyone who wants to
build a Boost release would now need to have your preferred build
system installed. Currently, we don't require anything else, as
Boost.Build is part of the release. So this would be a significant
regression in usability.

I understand. This is a bit of a vicious circle: Right now Boost is
always built as a whole, so lots of people do it. In a modular Boost
world, fewer people would build all of boost, as it's much easier to
build just the libraries people need.

I don't see it as much of a circle. As long as boost is monolithic, we need
to stick with one tool (b2 or cmake or whatever). After it splits into a
modular structure and no one needs/wants to build it all at once, then we
could open up other tools. I don't mind installing one or two
pre-requisites on my build machines, but if each library has their own
(conflicting?!?) requirements that'd get un-workable.

However, just because we go modular, doesn't mean that we should throw open
the door to each library maintainer doing whatever they want. It might not
be ideal for each library, but there is some benefit to standardizing on
tools. Other organizations put requirements on disparate project. For a
long time (not sure if this is still the case) the Apache project required
all its member projects to use SVN for source control, and those projects
are a lot less homogeneous than ours.

Tom

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

[Stefan Seefeld via Boost]

Post by Tom Kent via Boost
On Mon, Jun 19, 2017 at 2:57 PM, Stefan Seefeld via Boost <

Post by Stefan Seefeld via Boost

Post by Peter Dimov via Boost

What would it take for Boost to support individual libraries to be
built with anything else ?

In what scenario? Standalone, or as part of the Boost release?

Both, as the goal is not to add more infrastructure, but to replace it.

Post by Peter Dimov via Boost
If standalone, it's up to you to support whatever you like.
If as part of the release, this would mean that everyone who wants to
build a Boost release would now need to have your preferred build
system installed. Currently, we don't require anything else, as
Boost.Build is part of the release. So this would be a significant
regression in usability.

I understand. This is a bit of a vicious circle: Right now Boost is
always built as a whole, so lots of people do it. In a modular Boost
world, fewer people would build all of boost, as it's much easier to
build just the libraries people need.

I don't see it as much of a circle. As long as boost is monolithic, we need
to stick with one tool (b2 or cmake or whatever). After it splits into a
modular structure and no one needs/wants to build it all at once, then we
could open up other tools. I don't mind installing one or two
pre-requisites on my build machines, but if each library has their own
(conflicting?!?) requirements that'd get un-workable.

I agree. So, please consider my use-case of "I want to use my own build
tool" purely as an illustration of why modularization is useful.
The main goal of this modularization proposal remains to break the build
process up, so a top-level `./b2` invocation would do little more than
iterate over all Boost libraries and invoke some build command there
(the details of which remaining to be determined).

Post by Tom Kent via Boost
However, just because we go modular, doesn't mean that we should throw open
the door to each library maintainer doing whatever they want. It might not
be ideal for each library, but there is some benefit to standardizing on
tools. Other organizations put requirements on disparate project. For a
long time (not sure if this is still the case) the Apache project required
all its member projects to use SVN for source control, and those projects
are a lot less homogeneous than ours.

Understood. One reason why I'm writing this proposal is also to question
whether the level of homogeneity that we currently have (and require) is
actually enforcible (or even desirable) for a project the size of Boost.
It may well have been the right choice when Boost only consisted of a
handful of libraries. But nowadays, the question is at least worth being
asked again.

Stefan

[Gary Furnish via Boost]

I'd be happy with boost easily supporting some way of cross compiling
that does not involve undocumented features and writing a bunch of
custom jam files that show up in random locations. I'd be ecstatic if
there was a way to integrate it with an existing cmake build system.
Heck, I'd be happy with a clearly defined procedure so that I could
write a build file for some libraries. Some libraries build pretty
easily (python), some are so difficult and pull in so many
dependencies that it is faster to rewrite the code then figure out how
to install it independently (Log). I don't see as a user why this is
such a hard ask. If a tool like cmake can't be easily supported for
individual libs then the build system is quite frankly unmaintainable.
Is cmake such an exotic today that supporting it as an option is
prohibitive?

On Mon, Jun 19, 2017 at 1:48 PM, Peter Dimov via Boost

Post by Peter Dimov via Boost

What would it take for Boost to support individual libraries to be built
with anything else ?

In what scenario? Standalone, or as part of the Boost release?
If standalone, it's up to you to support whatever you like.
If as part of the release, this would mean that everyone who wants to build
a Boost release would now need to have your preferred build system
installed. Currently, we don't require anything else, as Boost.Build is part
of the release. So this would be a significant regression in usability.
_______________________________________________
http://lists.boost.org/mailman/listinfo.cgi/boost

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

[Chris Glover via Boost]

Post by Stefan Seefeld via Boost
I also want to be able to pick my own build (etc.) tools, not in
addition to Boost.Build, but instead of it. I understand that right now
that's not supported, which is why I'm writing this proposal. What would
it take for Boost to support individual libraries to be built with
anything else ? What requirements would that "anything" have to meet,
and how would it interact with the existing infrastructure to work ? Is
that such a strange request ?

I strongly disagree with allowing that to happen.

I'm a fan of the current model of distributing boost because of how I've
seen it distributed at large companies.

Every company I have worked at has maintained their own custom, internal
build system. To integrate boost into any of those systems has been pretty
easy -- it involves writing a script to bootstrap, invoke b2, copy headers
and libs and possibly generate some synthetic targets to let the custom
build system know how to find boost.

In some cases I've had to modify boost build to support weird proprietary
compiler variants -- which was a pain -- but imagine if every library was
doing their own thing? Now in order to use your library, I need to get
approval for the licence on the build system, then I need to make the build
system work and integrate it with whatever our custom setup is. Then
possibly I need to patch it to work with whatever special compiler we're
currently using, and I need to potentially do that more than once.

For me, the simplicity of monolithic (or at least unified) boost is worth a
huge amount and I would really hate to lose that. I feel the same about the
licensing issue that came up a few weeks ago for the same reasons --
getting approval to use and distribute boost at a large company is vastly
simpler if things are consistent.

-- chris

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

[John Maddock via Boost]

Post by Stefan Seefeld via Boost
I also want to be able to pick my own build (etc.) tools, not in
addition to Boost.Build, but instead of it. I understand that right now
that's not supported, which is why I'm writing this proposal. What would
it take for Boost to support individual libraries to be built with
anything else ? What requirements would that "anything" have to meet,
and how would it interact with the existing infrastructure to work ? Is
that such a strange request ?

Absolutely anything at all? You cannot do integration testing if every
library uses something different. You can't even do a single build and
install everything.

IMO there does have to be a common build system for that stuff (whatever
that may be), if authors want to ship with some other build system as
well, then that's just fine too.

John.

---
This email has been checked for viruses by AVG.
http://www.avg.com


_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

[Stefan Seefeld via Boost]

Post by John Maddock via Boost

Post by Stefan Seefeld via Boost
I also want to be able to pick my own build (etc.) tools, not in
addition to Boost.Build, but instead of it. I understand that right now
that's not supported, which is why I'm writing this proposal. What would
it take for Boost to support individual libraries to be built with
anything else ? What requirements would that "anything" have to meet,
and how would it interact with the existing infrastructure to work ? Is
that such a strange request ?

Absolutely anything at all?

(I'm not sure I understand what you mean. I'm specifically asking about
requirements that would restrict that "anything". So no, not absolutely
anything.)

Post by John Maddock via Boost
You cannot do integration testing if every library uses something
different. You can't even do a single build and install everything.

I think that's part of my point: At this point in time, who actually
needs the entirety of Boost built and installed as a single entity,
other than by habit ? There are so many different libraries, targeting
different audiences. Is there anybody using all of them ? Would it
really hurt anyone if they had to install Boost.MPI, Boost.Compute, and
Boost.Python (to name a few domain-specific ones) separately ?

Post by John Maddock via Boost
IMO there does have to be a common build system for that stuff
(whatever that may be), if authors want to ship with some other build
system as well, then that's just fine too.
John.

Stefan

[Rene Rivera via Boost]

On Tue, Jun 20, 2017 at 7:02 AM, Stefan Seefeld via Boost <

Post by Stefan Seefeld via Boost
I think that's part of my point: At this point in time, who actually
needs the entirety of Boost built and installed as a single entity,
other than by habit ? There are so many different libraries, targeting
different audiences. Is there anybody using all of them ? Would it
really hurt anyone if they had to install Boost.MPI, Boost.Compute, and
Boost.Python (to name a few domain-specific ones) separately ?

There have been a number of people who've expressed the experience and need
to *only* use Boost as a single entity in precisely these cmake/modular
threads. Who I would point out I've never seen post before. So it tells you
something about how strong their position is.

[Stefan Seefeld via Boost]

Post by Rene Rivera via Boost
On Tue, Jun 20, 2017 at 7:02 AM, Stefan Seefeld via Boost <

Post by Stefan Seefeld via Boost
I think that's part of my point: At this point in time, who actually
needs the entirety of Boost built and installed as a single entity,
other than by habit ? There are so many different libraries, targeting
different audiences. Is there anybody using all of them ? Would it
really hurt anyone if they had to install Boost.MPI, Boost.Compute, and
Boost.Python (to name a few domain-specific ones) separately ?

There have been a number of people who've expressed the experience and need
to *only* use Boost as a single entity in precisely these cmake/modular
threads. Who I would point out I've never seen post before. So it tells you
something about how strong their position is.

Yes, of course ! I do understand the advantage of Boost being a single
entity. And to some there is just that advantage, as they don't have to
deal with the disadvantage(s). So in the end it's a balancing act where
we have to weigh the different arguments.

I still think we are getting ahead of ourselves, as my proposal wasn't
(and still isn't) about replacing build systems (even though it is
definitely motivated by that option), but it's about modularizing the
process, to make it easier to build components (i.e., libraries)
stand-alone.

Once that is possible, and once people start to actually do build (and
use) components separately, our perspective on what Boost is and how it
is being used may change.

Stefan

[Peter Dimov via Boost]

Post by Peter Dimov via Boost
Why is this so important for you? The difference is just one
superproject shell and one tools/build. Is this a matter of principle,
or are there technical reasons?

Both. I want to control the environment in which my library is being built
(, tested, etc.), and thus I don't want to fetch that environment from a
repository and build it on-the-fly.

...

I think it's perfectly normal that on some platforms the default (system)
version of a package is too old, so a newer version needs to be pulled in
from another repo ("testing", perhaps ?).

So you consider it perfectly normal to pull boost.build 1.64 from some
"testing" repo, but completely unacceptable to checkout the boost-1.64.0 tag
of github.org/boostorg/build?

I'm sorry, but this makes no sense to me.


_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

[paul via Boost]

 I know about these tricks (I use something similar to set up my 
Boost.Python CI environment), but consider them hacks as they don't
really 
* I want to be able to build a given Boost library stand-alone, with 
nothing but the library's repo being checked out. (In other words: 
prerequisite Boost components should be assumed pre-installed.)

Why is this so important for you? The difference is just one superproject 
shell and one tools/build. Is this a matter of principle, or are there 
technical reasons?
Anyway.
    sudo yum install boost-jam
    sudo yum install boost-build
    git clone --depth=1 https://github.com/boostorg/python
    cd python
    touch Jamroot
    bjam test
The first error here is
    IMPORT error: rule "requires" unknown in module 
"../../config/checks/config"
because we don't have Boost.Config checked out as a sibling here.

Instead of hard-coding paths to the build modules, it would be nice if it
searched for the modules instead, and then it could fallback on the hardcoded
paths when the search fails.

Ideally, instead of inventing a search algorithm to find the modules,
pkgconfig could be used here. So when boost config is installed, the .pc would
add a variable that is the location of its bjam files for consumption:

prefix=<install-location>
bjam_dir=${prefix}/bjam_files
Name: boost_config

And then you can call `pkg-config boost_config --variable=bjam_dir` and it
will give the directory of the bjam modules installed with boost_config.


_______________________________________________
Unsubscribe & other changes: http://lists.boos

[Peter Dimov via Boost]

Post by paul via Boost

Post by Peter Dimov via Boost
The first error here is
IMPORT error: rule "requires" unknown in module
"../../config/checks/config"
because we don't have Boost.Config checked out as a sibling here.

Instead of hard-coding paths to the build modules, it would be nice if it
searched for the modules instead, and then it could fallback on the
hardcoded paths when the search fails.
Ideally, instead of inventing a search algorithm to find the modules,
pkgconfig could be used here. So when boost config is installed, the .pc
would add a variable that is the location of its bjam files for

...

This won't work. The system Boost.Config installation on CentOS 7 is just
the headers, as part of (yum) package boost. There are no bjam files
anywhere to be found, no pkgconfig, no boost_config module.


_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

[Robert Ramey via Boost]

Post by Stefan Seefeld via Boost
* I want to be able to build a given Boost library stand-alone, with
prerequisite Boost components should be assumed pre-installed.)
* I want to be able to choose the build (, test, etc.) infrastructure
used for my library, and make sure that when Boost is built as a whole,
that build infrastructure is used.

Hmmmm - I'm thinking I doing that already. Here's what I do:

a) I have a modular boost clone on my machine - set to master branch.
b) I also have in the same tree libraries which I'm working on with the
branch set to develop. These might be libraries already in boost, like
boost.radional or boost.serialization. They also might be totally new
libraries/applications which are not in boost at all.
c) working from my shell move the directory which interests me. Its
currently boost/libs/safe_numerics/test. But it could be anywhere which
has a Jamfile.v2
d) then I invoke b2* which builds anything dependent and the then the
target in the local Jamfile.v2
e) So then I've got exactly what I need with pretty no hassle - except
getting the switches to bjam right. This last is a hassle as I test
with various toolsets.

* actually rather than invoking b2 directly, I invoke
../../../library_status.sh which runs b2 and then produces my own very
cool html global table of all the tests I've run by compiler, build
variant, link variant, etc.

Sooooo - I'm not getting what we're missing here.

And I'm getting what the CMake advocates want either. Actually I'm not
understanding what anyone (but me) wants which we don't already have
except an easier way to do what we're already doing.

BTW I do use CMake because I like to use IDE for edit, test etc. Using
IDE without this was a painful maintainence nightmare. Fortunately
making a CMake to build my IDE is pretty simple. You can see what I did
in the github develop branch of safe_numerics or serialization library.
The Jamfile.v2 scripts are actually even simpler - just a list of tests
to run.

Robert Ramey



_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

[Andrey Semashev via Boost]

Post by Peter Dimov via Boost
Releasing separately from Boost means that you now have two
Boost versions to communicate: the version of Boost.Python, and the
version of the preinstalled Boost against which it was built (I built,
f.ex. libboost-python-1.65.0-1.53.0 in my CentOS 7 case.)
This consequently affects your downstream dependencies, which also have
to include these two versions, in addition to their own. So when a
library X depends on Boost.Python and it's compiled against Boost.Python
1.65.0-1.53.0, its version would now be X-1.17-1.65.0-1.53.0. This is
not compatible with Y-2.1-1.63.0-1.53.0, even though both are built in
the same CentOS 7 environment.

I think, you unnecessarilly complicate things. Yes, naturally lower
level dependencies define the effective set of software that is required
for the upper level software to build, run and be tested. But it doesn't
mean those dependencies need to be reflected in version numbers (they
don't) or that the upper level software doesn't work with different
versions of the dependencies. Normally, when you release a piece of
software, you declare the minimum versions of the prerequisites -
preferably, the oldest versions that were successfully tested.

IMHO, formal dependency management is a task related to packaging, which
is specific to the packaging system. I don't think Boost should be doing
this work beyond documenting the dependencies. Let packagers deal with
the technical part of enforcing these dependencies with the means they
have in their disposal.

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

[degski via Boost]

When I speak of dependencies I am not just speaking of library X depending
on other libraries A, B, and C etc. I am also speaking of library X
depending on particular versions of library A, B, C etc. But since the only
versioning system Boost has is a single version number for a Boost release,
and since their is no way a library can check even that single version
number of a Boost release either at compile or run-time, Boost libraries
have no way to check versioning of other individual Boost libraries on
which a library may depend.
If you say, I am going to distribute my library X with particular releases
of library A, B, and C etc. with which I know my library X will work
correctly you then have end-users of your library who may have multiple
copies of libraries X, A, B, and C etc. on their systems, ...

and:

Robert Ramey wrote: "In order to distribute boost as individual libraries
as opposed to a monolithic set, individual library versioning will sooner
or later have to be adopted."

This is exactly how rust's Cargo works. Developers can specify the version
or version+ against which his library should be built. I'm not claiming
Cargo is perfect, but it's really pretty good at this and creates complete
transparency as to what the dependencies are. Having multipple versions
around is a consequence of this though. Cargo just works! Bar errors that
did not came out in testing, this guarantees stability for every library
individually, which in its' turn guarantees overall stability. This is
pretty neat!

Let's be realistic, this is a real problem which only some sort of Boost
individual library versioning system for starters can hope to solve.

The build system should address/include the versioning, Cargo should be
taken as a model, me thinks.

degski