Arch Linux Cloud Images (virtualbox and Qemu)

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Arch Linux Cloud Images (virtualbox and Qemu)

arch dev mailing list
Hello everybody,
Some months ago Bartlomiej had the awesome idea to generate qemu and
virtualbox images as well. Afterwards we had a small discussion about
this topic in #archlinux-projects. I would like to refresh this
discussion and move it to this mail thread.

So, our vagrant boxes are building automatically and without issues for
a few months now. I could just generate virtualbox and qemu images as
side products monthly, but I have no idea how we would release them.

Another big topic is security. The vagrant images are at the moment not
signed and seems like vagrantcloud doesn't support this either. This
doesn't mean that we shouldn't sign our qemu or virtualbox images.
We could just generate an automated cloud image signing key (only for
this purpose) of course and automatically sign the images with that key.
Problem with this is: If our build server ever get pwned the person will
have these keys for signing cloud images as well. Any opinion about
this?

In my humble opinion I would be fine with a signing key that follows
only this particular purpose. Another idea would be to move the whole
production process to a physical server behind a firewall that is
physical accessible by one or more Arch Linux Developers.

I really would like to push this forward. Any comments?

Chris

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Automatic Signing of ISOs, pacman databases and everything else (was: Arch Linux Cloud Images (virtualbox and Qemu))

arch dev mailing list
On 13.05.2018 22:47, Christian Rebischke via arch-dev-public wrote:
> We could just generate an automated cloud image signing key (only for
> this purpose) of course and automatically sign the images with that key.
> Problem with this is: If our build server ever get pwned the person will
> have these keys for signing cloud images as well. Any opinion about
> this?

We had that discussion some years ago about signing our pacman
databases. I mostly remember that we didn't reach a consensus, but you
might want to search the archives for details. At some point there was a
proposal to have a dedicated signing host that is well protected and
receives files and then returns the signature. I'm not sure if that was
turned down or if there was simply nobody to work on this. Does anyone
remember that?

I think this would be a viable option for us. We could also implement
some form of rate limiting and sanity checks to ensure we only sign
things that we want to sign. For example, only one ISO can be signed per
month and the request must come from a specific IP. I probably won't do
any implementation, but I'd offer to provide feedback and design help if
someone wants to work on this. Assuming we first agree that we want to
do it this way.

Florian


signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Automatic Signing of ISOs, pacman databases and everything else (was: Arch Linux Cloud Images (virtualbox and Qemu))

arch dev mailing list
Le 15/05/2018 à 17:25, Florian Pritz via arch-dev-public a écrit :

> On 13.05.2018 22:47, Christian Rebischke via arch-dev-public wrote:
>> We could just generate an automated cloud image signing key (only for
>> this purpose) of course and automatically sign the images with that key.
>> Problem with this is: If our build server ever get pwned the person will
>> have these keys for signing cloud images as well. Any opinion about
>> this?
> We had that discussion some years ago about signing our pacman
> databases. I mostly remember that we didn't reach a consensus, but you
> might want to search the archives for details. At some point there was a
> proposal to have a dedicated signing host that is well protected and
> receives files and then returns the signature. I'm not sure if that was
> turned down or if there was simply nobody to work on this. Does anyone
> remember that?
>
> I think this would be a viable option for us. We could also implement
> some form of rate limiting and sanity checks to ensure we only sign
> things that we want to sign. For example, only one ISO can be signed per
> month and the request must come from a specific IP. I probably won't do
> any implementation, but I'd offer to provide feedback and design help if
> someone wants to work on this. Assuming we first agree that we want to
> do it this way.
To me this is quite a good idea. :)

I had a bit more sophisticated design in mind, where the signing host
/retrieves/ the file to be signed (so that the connection is initiated
from it, not toward it) by having the filename added to some text file
on an other (almost?) dedicated host (so that having access to the hosts
where the DB/iso/whatever are built is not enough and vice-versa, see
just after), text file that the signing host would be watching a way or
another (but should be in an authenticated way). Of course you need to
restrict what kind of files can be retrieved from what host (like you
proposed for the request coming from a specified IP).

The goal of this setup is to have no open port on the signing host,
requiring physical/IPMI access to it to make any change.

But maybe that does not bring much more than your setup, while adding
much more complexity…

Just as you, I cannot help on implementing, but I can offer ideas and
design feedback if anyone want to take this task in charge.

Regards,
Bruno


signature.asc (499 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Automatic Signing of ISOs, pacman databases and everything else (was: Arch Linux Cloud Images (virtualbox and Qemu))

Jelle van der Waa-2
On 05/15/18 at 05:43pm, Bruno Pagani via arch-dev-public wrote:
> Le 15/05/2018 à 17:25, Florian Pritz via arch-dev-public a écrit :

Just going to necro-bump this thread, since we didn't arrive at a
conclusive descision.

>
> > On 13.05.2018 22:47, Christian Rebischke via arch-dev-public wrote:
> >> We could just generate an automated cloud image signing key (only for
> >> this purpose) of course and automatically sign the images with that key.
> >> Problem with this is: If our build server ever get pwned the person will
> >> have these keys for signing cloud images as well. Any opinion about
> >> this?
> > We had that discussion some years ago about signing our pacman
> > databases. I mostly remember that we didn't reach a consensus, but you
> > might want to search the archives for details. At some point there was a
> > proposal to have a dedicated signing host that is well protected and
> > receives files and then returns the signature. I'm not sure if that was
> > turned down or if there was simply nobody to work on this. Does anyone
> > remember that?
> >
> > I think this would be a viable option for us. We could also implement
> > some form of rate limiting and sanity checks to ensure we only sign
> > things that we want to sign. For example, only one ISO can be signed per
> > month and the request must come from a specific IP. I probably won't do
> > any implementation, but I'd offer to provide feedback and design help if
> > someone wants to work on this. Assuming we first agree that we want to
> > do it this way.
I believe this solution is the way to go.

> To me this is quite a good idea. :)
>
> I had a bit more sophisticated design in mind, where the signing host
> /retrieves/ the file to be signed (so that the connection is initiated
> from it, not toward it) by having the filename added to some text file
> on an other (almost?) dedicated host (so that having access to the hosts
> where the DB/iso/whatever are built is not enough and vice-versa, see
> just after), text file that the signing host would be watching a way or
> another (but should be in an authenticated way). Of course you need to
> restrict what kind of files can be retrieved from what host (like you
> proposed for the request coming from a specified IP).
>
> The goal of this setup is to have no open port on the signing host,
> requiring physical/IPMI access to it to make any change.
>
> But maybe that does not bring much more than your setup, while adding
> much more complexity…
>
> Just as you, I cannot help on implementing, but I can offer ideas and
> design feedback if anyone want to take this task in charge.
That sounds rather complicated, since we also wants this for the repo
db as well. I wonder if we use the proposed method but restrict access
not only source ip but also on the user who can make the request?

On a seperate note,  I don't believe the signing issue is new I know
that Fedora and OpenSuSe have both signing solutions. For the OpenSuse
Build Service, they have a daemon called obs-signd. [1]

Their solution is a sperate machine with a port open for their
signing daemon. I'm not sure how they resolve the don't sign any
arbitrary file problem.

For Fedora I couldn't find any information, I've reached out to a Fedora
Dev for some more information. The only thing I can find is a proposal. [2]

Maybe we should create a wiki page for signing the repository DB and
ISO's. So we can list all the benefits and downsides along with the
threat vector.

[1] https://en.opensuse.org/openSUSE:Build_Service_Signer
[2] https://fedoraproject.org/wiki/Koji_Build_Autosign_Proposal

--
Jelle van der Waa

signature.asc (499 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Automatic Signing of ISOs, pacman databases and everything else (was: Arch Linux Cloud Images (virtualbox and Qemu))

arch dev mailing list
In reply to this post by arch dev mailing list
On Tue, May 15, 2018 at 05:25:07PM +0200, Florian Pritz via arch-dev-public <[hidden email]> wrote:

> On 13.05.2018 22:47, Christian Rebischke via arch-dev-public wrote:
> > We could just generate an automated cloud image signing key (only for
> > this purpose) of course and automatically sign the images with that key.
> > Problem with this is: If our build server ever get pwned the person will
> > have these keys for signing cloud images as well. Any opinion about
> > this?
>
> We had that discussion some years ago about signing our pacman
> databases. I mostly remember that we didn't reach a consensus, but you
> might want to search the archives for details. At some point there was a
> proposal to have a dedicated signing host that is well protected and
> receives files and then returns the signature. I'm not sure if that was
> turned down or if there was simply nobody to work on this. Does anyone
> remember that?
>
> I think this would be a viable option for us. We could also implement
> some form of rate limiting and sanity checks to ensure we only sign
> things that we want to sign. For example, only one ISO can be signed per
> month and the request must come from a specific IP. I probably won't do
> any implementation, but I'd offer to provide feedback and design help if
> someone wants to work on this. Assuming we first agree that we want to
> do it this way.
After the topic was raised again in #archlinux-dev I've created a POC
script which you can view here:
https://git.server-speed.net/users/flo/alass/tree/alass

There are still various open steps to actually put this to use:
 - Create an ansible role that deploys the script + configures an SSH
   user for it (including GPG key)
 - Determine if and how we want to extend the verification logic in
   allowed_to_sign(). We may want to verify the content of the tarball
   (if the signed file is a package), ratelimit the requests or perform
   some other checks. Right now it verifies the file name, but that is
   not part of the signature, so it can easily be faked.  We can
   implement some restrictions via SSH (limit the remote IP to e.g.
   orion.archlinux.org and obviously use an SSH key for authentication),
   but any checks of the data have to be performed in the script.
 - Once we are happy with it all, we need to set up a dedicated host
   (I'd suggest a small vm @hetzner), create a new GPG key and get it
   signed by our master keys. Then we can start automatically signing
   things.

If anyone wants to help with any of these steps (especially the
verification logic) please post here or talk to me on IRC. If you have
ideas what/how to verify things, also please post and keep in mind that
we should aim for simple checks since those are less likely to result in
security issues. This script essentially receives untrusted input.

Florian

signature.asc (849 bytes) Download Attachment