[14:07:22] [connected at Thu Feb 27 14:07:22 2020]
[14:07:33] [I have joined #xf-bod]
[14:08:29] <mdnavare> I was  out last 2 weeks so couldnt reply on the Gdansk thread
[14:08:42] <mdnavare> I will do that today
[14:09:11] <danvet_> ok so we're starting for real now?
[14:09:36] <Lyude> ok! hi-agenda for today (hopefully I'm not missing anything) - VESA status, XDC2019 sponsor invoicing, sponsored travel receipts for XDC2019, x.org/freedesktop.org domain status, requesting ledger data from SPI, elections, clo9ud hosting for freedesktop.org, and GSoC/EVoC
[14:09:38] <Lyude> danvet_: yeah
[14:10:28] <tlwoerner> sounds good
[14:11:01] <danvet_> need to make sure we get around to the fd.o cloud hosting stuff
[14:11:35] <Lyude> for vesa stuff: still haven't heard a response yet but didn't get a chance to poke anyone since I've been on PTO since wednesday (was supposed to be back on monday but that's another story), same for fd.o domain stuff but keithp might know more on that
[14:12:21] <Lyude> danvet_: do we have any updates regarding that btw?
[14:12:49] <keithp> I haven't heard anything about the fd.o domain stuff fwiw
[14:13:01] <hwentlan> i think we'll need to discuss state and next steps a bit. not sure if everyone followed the thread with daniels and danvet_
[14:13:17] <danvet_> Lyude, maybe still get through the other topics quickly?
[14:13:19] <danvet_> this might take long
[14:13:28] <Lyude> danvet_: yeah sure thing
[14:13:35] <danvet_> meanwhile everyone can make sure they caught up on the long thread on board@ :-)
[14:13:36] <tlwoerner> good thing we started early...
[14:14:05] <Lyude> anholt: you around for sponsor invoicing? and while we wait for them to respond, ledger data stuff from SPI: haven't heard anything back from them yet
[14:14:32] <anholt> arm has maybe finally succeeded in getting a payment moving in their process
[14:14:32] <danvet_> do we need to kick spi harder on this invoicing?
[14:14:42] <danvet_> I can kick intel, but not without being sure that the problem is on intel's side
[14:14:45] <anholt> intel is still blocked on intel
[14:14:57] <anholt> there is definitely absolutely a problem on intel's side
[14:15:01] <Lyude> danvet_: yeah we might want to do that with intel
[14:15:02] <danvet_> anholt, what's amiss?
[14:15:12] * danvet_ happy to kick down doors
[14:15:17] <danvet_> just need to know which one :-)
[14:15:28] <anholt> danvet_: they're insisting on going through a vendor payment portal thing that is supposed to send out an invite email, spi never gets the invite email, nobody on the intel side can debug anything.
[14:15:42] <mdnavare> danvet_: I know an email was sent to our VP in this regards as well
[14:15:48] <anholt> you should probably go kick them to just ditch doing this as a vendor thing and put it on a credit card
[14:15:52] <danvet_> anholt, ok I'll crack that, thx
[14:16:10] <Lyude> anholt: any sponsored travel receipts for XDC 2019 that need approving?
[14:16:45] <anholt> nope, the one troublesome one I sorted out with spi treasurer based on our previous vote
[14:16:51] <anholt> nobody else has replied in the meantime
[14:16:57] <Lyude> alright
[14:17:12] <Lyude> tlwoerner: anything on the GSoC/EVoC side?
[14:17:23] <tlwoerner> Lyude: yes, thank you
[14:17:32] <tlwoerner> X.org has been accepted into GSoC 2020
[14:17:37] <tlwoerner> 200 orgs this year, including 30 new ones
[14:17:43] <tlwoerner> we're now in the "get mentors, update projects list" phase
[14:17:50] <tlwoerner> students start applying March 16
[14:17:54] <bryce> congrats
[14:17:57] <tlwoerner> i need to:
[14:18:05] <tlwoerner> - start sending emails to the various lists
[14:18:09] <tlwoerner> - there are already some rumblings on the lists
[14:18:14] <tlwoerner> - email Sebastian from last year
[14:18:19] <tlwoerner> bryce: thanks!
[14:18:32] <tlwoerner> that is all (i think)
[14:18:50] <tlwoerner> we have 1 mentor so far signed up
[14:19:07] <mdnavare> tlwoerner: I have already accepted and fille dout the gsoc mentor agreement
[14:19:10] <danvet_> Lyude, totally forgot: agenda += elections, xdc20 sponsors
[14:19:24] <tlwoerner> mdnavare: excellent! that's 2 mentors :-)
[14:19:26] <Lyude> danvet_: yeah I was about to start asking about elections :P
[14:19:48] <mdnavare> tlwoerner: Yes I am on the gsoc mentors mailing list now
[14:19:52] <Lyude> but i did forget about 2020 sponsors
[14:20:05] <Lyude> danvet_: any update on either of those two things?
[14:20:08] <tlwoerner> mdnavare: great!
[14:20:35] <danvet_> Lyude, elections: still blocked on https://gitlab.freedesktop.org/freedesktop/freedesktop/issues/244 I pinged daniels a few times
[14:20:36] <mdnavare> tlwoerner: Do you need any helpin completing application instructions for students etc?
[14:20:42] <danvet_> but very gently, he's overloaded
[14:21:01] <danvet_> xdc20 sponsors: I think we're unblocked now with the private board@ thread, I'll restart that next week
[14:21:02] <tlwoerner> mdnavare: i think we're all good, we'll use the same procedures as the last couple years. they seem to work well enough
[14:21:23] <tlwoerner> i just need to wrangle up more mentors and make sure the ideas list is up-to-date
[14:21:26] <danvet_> oh also, xdc20 website: I sent a mail, I think samuelig volunteered to take care of that and get it going
[14:21:39] <danvet_> people are asking questions about when/where xdc exactly is
[14:22:39] <Lyude> danvet_: we ready to move on to the cloud hosting stuff?
[14:23:11] <danvet_> yeah I think so
[14:23:18] <danvet_> should I do a super short summary?
[14:23:34] <Lyude> if you want, we've got the whole rest of the meeting to discuss this
[14:23:46] <Lyude> that might help though since I haven't seen any of the thread yetr
[14:23:47] <hwentlan> danvet_, daniels, thanks for covering this copy in the email thread. please summarize it quickly
[14:23:51] <danvet_> ok so good news: gitlab.fd.o is seeing massive uptick in use, people like CI
[14:23:57] <danvet_> bad news: it's breaking our bank
[14:24:13] <danvet_> my rough estimate, taking some reasonable growth into account
[14:24:19] <danvet_> is 75k USD this year
[14:24:33] <danvet_> looking at Jan spi report, we have 75K in the bank right now
[14:24:49] <danvet_> daniels thinks it's going to be less, but he's maybe a bit too optimistic
[14:25:04] <danvet_> if we assume growth stops there, it's like 95k USD / year
[14:25:26] <danvet_> now there's a really long thread on board@ about what to do and where to get sponsors for this
[14:25:35] <danvet_> big thing is that most of the growth is because of CI
[14:25:48] <danvet_> not runners, but runners who upload/download data to gitlab.fd.o
[14:26:05] <danvet_> we estimate that with CI functionality disabled the bill is 30k USD/year
[14:26:06] <anholt> I've looked into some of the usage in a bit more detail, and can definitely confirm that CI can be attributed to most of our costs.
[14:26:27] <danvet_> and yeah that's a very rough estimate, we'd need better numbers
[14:26:31] <hwentlan> anholt, danvet_, is most of the cost the CI runners or the bandwidth?
[14:26:39] <anholt> hwentlan: this is explicitly not counting runners at all.
[14:26:41] <danvet_> hwentlan, 0 runners in that estimate
[14:26:43] <anholt> those are all sponsored by others.
[14:26:48] <danvet_> we killed all the X.org paid runners
[14:26:54] <anholt> but runners consume bandwidth moving containers and artifacts, and storage in storing containers and artifacts
[14:27:07] <danvet_> hwentlan, I think it's 90% bandwidth 10% storage or artifacts
[14:27:12] <danvet_> or even more loopsided
[14:27:19] <danvet_> we gc artifacts and build logs quite quickly
[14:27:22] <danvet_> and storage is cheap
[14:27:41] <hwentlan> siqueira showed me that aws and digitalocean have open source sponsorship programs we could apply for
[14:27:42] <anholt> danvet_: storage is pretty big too.  we're like 2/3 of our storage cost on artifacts and registry iirc, and storage and egress were basically evenly split in the top-level costs.
[14:27:55] <danvet_> anyway from an fd.o admin pov it's quite simple to disable CI complete and so pretty much instantly cut the costs
[14:28:02] <hwentlan> but i'm curious whether it'd be feasible to distribute storage/bandwidth across different providers
[14:28:10] <danvet_> anholt, hm I guess I looked at this wrong then
[14:28:14] <anholt> hwentlan: moving to another cloud provider is tremendously expensive in admin time costs, which we have only gracious volunteering from daniels for.
[14:28:17] <danvet_> gloud storage looked tiny
[14:28:19] <bryce> how is the search for sponsors going for this?
[14:28:41] <Lyude> i am poking red hat about it again now
[14:28:44] <anholt> I'm working on getting sponsorship from my employer, and notably they think they can get admin time sponsored next year.
[14:28:47] <hwentlan> anholt, thanks. i thought so. so that's pretty much a no-go then
[14:28:48] <bryce> and are you looking only for financial sponsors, or also sponsors that can provide hosting as well?
[14:28:53] <anholt> that leaves us in a bind for this year.
[14:29:02] <Lyude> going to start pulling out the blue "we have the money though" card and hope that gets us somewhere :S
[14:29:08] <daniels> [re elections: I'll have it done on the weekend at the latest but jetlag and other commitments have made it difficult to get done now]
[14:29:11] <anholt> bryce: hosting on another provider costs admin time, unless fd.o has a paid admin I don't think that's really on the table.
[14:29:12] <danvet_> bryce, see thread, hosting is awkward
[14:29:19] <danvet_> daniels, thx very much
[14:29:36] <Lyude> bryce: I think we're looking for financial sponsors and even better, someone to sponsor an actual sysadmin
[14:29:41] <danvet_> bryce, except if it comes in the form of google cloud credits
[14:30:16] <danvet_> so anyway the trouble is that if we want some cash flow reserves for xdc and bare-bones gitlab.fd.o and all that for 1 year
[14:30:26] <danvet_> we'd need to close gitlab CI somewhere in May or June this year
[14:30:32] <danvet_> assuming no money comes in
[14:30:51] * anholt put together a sponsorship proposal and reasoning to chromeos, we'll see where that goes.
[14:31:05] <danvet_> imo board needs to take a decision on this (how much cash flow reserves and all that) now and communicate it
[14:31:29] <danvet_> otherwise (given our usual decision latency) we might burn a real hole into x.org's bank account
[14:31:36] <hwentlan> should we have a public sponsorship proposal somewhere on x.org or freedesktop.org?
[14:31:45] <danvet_> plus everyone's surprised if we don't announce this as early as possible I think
[14:31:53] <bryce> +1 to putting as much on pause as needed to stabilize the finances
[14:31:55] <hwentlan> something that we can point companies to in order to show why we need money and what the benefits are?
[14:32:23] <Lyude> have we also looked at poking the kernel-ci project and seeing if maybe they'd be interested in helping since this sort of affects them too?
[14:32:37] <Lyude> also, rh is wondering which other companies are currently sponsoring us if there are any
[14:32:39] <danvet_> bryce, we've done everything that doesn't disrupt services already
[14:32:40] <anholt> Lyude: not seeing much overlap with kernel-ci here
[14:32:45] <danvet_> this would be about the stop loss going forward
[14:33:06] <hwentlan> +1 on clear message to community that CI is burning through cash and needs to be paused/disabled in may if we don't have new sponsorships by then
[14:33:07] <danvet_> Lyude, yeah the problem isn't runners themselves, it's the bw to those runners
[14:33:22] <danvet_> so kernel-ci is part of the problem here (since some of the mesa runners are in kernel-ci afaiui)
[14:33:27] <anholt> I do think we need to post something soon and to raise awareness
[14:33:32] <Lyude> danvet_: i just meant they might be able to help with finding sponsors
[14:34:08] <Lyude> anyway +1 on any announcements we need to make on this
[14:34:11] <daniels> [my take on the above is that medium-term the admin load is enough to necessitate making it a paid role, but if in the short term the answer is moving to AWS then I can make that happen sometime before summer]
[14:34:12] <danvet_> oh I know plenty of contacts for sponsors, that's not the big problem I think
[14:34:27] <samuelig> +1 on announcement as well.
[14:34:31] <anholt> we have an important question for the google proposal: is the board OK with google sponsoring an fd.o sysadmin, effectively of their choice (it would be someone at collabora)?
[14:34:42] <danvet_> daniels, yeah I think we should also put the "we'll need a paid admin medium term" into that announcement
[14:34:45] <mdnavare> +1 for posting on X.org site and public announcement at XDC etc
[14:34:54] <daniels> [what I'm worried about is splitting between services and chopping and changing. one service is tractable but needing to mix and shift is too much for my availability.]
[14:35:00] <Lyude> anholt: +1 from me, that would be awesome
[14:35:19] <danvet_> daniels, yeah it's definitely no good at all given that the move to cloud was meant to solve these kind of things
[14:35:27] <anholt> iirc we had something about the board needing to approve specific new admins, which in this case we wouldn't get to.
[14:35:43] <danvet_> anholt, +1 too, there's some issues with company sponsored admins
[14:35:55] <danvet_> but realistically we won't be able to get that going with spi anytime soon I think
[14:36:02] <samuelig> anholt, +1 too.
[14:36:04] <danvet_> so stop-gap this would be awesome
[14:36:22] <hwentlan> +1 on google-sponsored admin (through collabora or otherwise)
[14:36:46] <danvet_> Lyude, so is there clear announcement about fd.o situation already passed?
[14:37:01] <danvet_> I could volunteer for drafting that, and then get it checked by hwentlan and send it out
[14:37:02] <hwentlan> though since this is something we haven't done before... are there implications to this that we're not considering but might regret in a year or two?
[14:37:06] <Lyude> could they not just have us approve the admin they choose? it's not really a definite since in theory we could end up saying no, but I don't think anyone here would unless there's an extremely good reason
[14:37:08] <anholt> (I'm +1 on one of our employers handling it, because I think getting our board into the business of employment more directly, particularly when the likely targets are members of our community, is even more risky)
[14:37:11] <danvet_> just the notes in the minutes might be a bit terse
[14:37:29] <Lyude> danvet_: say that again? I'm not sure what you mean by already passed
[14:38:54] <danvet_> we have like 2 votes on different things going on here right now
[14:39:26] <Lyude> yeah i ran the meeting too fast I'm afraid :S
[14:39:29] <danvet_> there's the "google sponsored admin" from anholt and "clear announcement about fd.o and that we might need to cut CI by May/June" from me
[14:40:17] <Lyude> ok, +1 from me on both of those things, danvet_ if you wouldn't mind drafting a response that would be good since i've got a lot to catch up on today that built up over PTO
[14:40:50] <bryce> I'd prefer seeing the draft before voting on it, but no objections in concept
[14:41:17] <daniels> last thing I'll say - practically the big cloud providers are good right now. Google Cloud Storage (and AWS S3 / Azure Storage are interchangeable for this) are expensive, but in return for the cost you gain time. bare-metal providers like DO are cheaper in immediate cost but then you need admin time to manually monitor and provision more storage. I don't have that time right now. a more dedicated admin could make bare-metal practical 
[14:41:17] <daniels> but it would take time and a longer-term commitment that they would be able to stay on top of it
[14:41:19] <Lyude> anholt, hwentlan, samuelig, gave +1 to the google sponsored employee, so we've got consensus there
[14:41:46] <hwentlan> +1 on announcement. would prefer it goes out sooner, rather than waiting until next meeting to vote on the actual text
[14:41:58] <anholt> hwentlan: agreed
[14:42:10] <danvet_> yeah I don't think we should wait
[14:42:16] <danvet_> bryce, happy to ping you before I send it
[14:42:27] <danvet_> also, it's all going to be summarized in the minutes anyway
[14:42:41] <danvet_> so phoronix article in like tomorrow latest
[14:42:42] <bryce> danvet_, you might leave the timeframe unspecified in case you need to adjust it forward or backward
[14:43:02] <Lyude> hwentlan, mdnavare, samuelig, myself and bryce gave +1 on the sponsorship announcement so we're approved on that
[14:43:15] <hwentlan> on the admin question i agree with anholt that it's better to have them sponsored by a company in our community, rather than employed by x.org directly
[14:43:15] <danvet_> bryce, I think qualifying it that it's assuming no surprises and no sponsoring income should be enough
[14:43:36] <danvet_> Lyude, ok I'll type something and ping hwentlan and bryce at least
[14:43:47] <danvet_> anyone else too ofc who wants to be involved
[14:44:26] <Lyude> alright, anything else we should go over before the meeting finishes?
[14:44:32] <Lyude> we've covered the agenda
[14:45:13] <mdnavare> danvet_: samuelig: I will respond to the gdansk thread today or do we already have consensus?
[14:45:31] <danvet_> mdnavare, see the thread, no more public mentioning :-)
[14:45:36] <danvet_> but yeah reply would be good
[14:45:39] <danvet_> same for bryce 
[14:45:42] <danvet_> then I think we have everyone
[14:45:53] <Lyude> alright - that's it for the meeting then everyone
[14:45:53] <mdnavare> ok cool
[14:46:02] <Lyude> thanks for coming!
[14:46:03] <samuelig> mdnavare, please reply :)
[14:46:09] <mdnavare> yes
[14:47:37] <Lyude> anholt: btw - can you get me a list of any sponsors xorg currently has that we're getting hosting money/resources from?
[14:47:56] <Lyude> i'm pushing red hat a bit harder this time with trying to get them to sponsor and that'd definitely help
[14:48:39] <anholt> Lyude: google OSPO was the only sponsor we had for these services.
[14:48:41] <Lyude> no guarantees but I might actually be able to get something since google's potentially providing us with a full time admin
[14:48:57] <anholt> (gitlab theoretically gave us some startup sponsorship, and I should try to reopen that discussion but it's died several times)
[14:57:40] <danvet_> Lyude, see thread
[14:57:52] <danvet_> it's in there somewhere
[14:58:08] <Lyude> danvet_: mhm, anholt answered it
[14:58:11] <danvet_> aside from the gitlab sponsoring, which isn't ongoing, was just a one-off last year
[14:58:14] <danvet_> Lyude, there's more
[14:58:19] <Lyude> ooooh ok
[14:58:21] <Lyude> i will take a look then
[14:58:27] <danvet_> we have a pile of CI runner sponsors too
[14:58:50] <danvet_> also more numbers
[15:00:22] <danvet_> hwentlan, bryce https://paste.debian.net/1132596/ quick draft
[15:02:56] <hwentlan> danvet_: lgtm
[15:02:58] <anholt> +1
[15:07:04] <danvet_> plan is to send this to members@ and usual mailing lists
[15:07:08] <danvet_> like xdc announcement
[15:07:23] <danvet_> does anyone have a gstreamer mailing list?
[15:07:37] <keithp> +1
[15:08:15] <danvet_> ah gstreamer-devel still seems to be a thing
[15:08:25] <danvet_> they haven't entirely abandoned that
[15:14:28] <danvet_> anholt, just looked at the .csv files from spi again
[15:14:49] <danvet_> only cloud storage I could find was ~7TB-months for ~150$
[15:15:00] <danvet_> we have more storage for the GCE's but I didn't count that
[15:15:12] <danvet_> and per daniels most of the CI stuff should be on cloud storage directly
[15:15:53] <danvet_> the gce ssd is sometwhat expensive, but kind small compared to the bw we burn down
[15:16:08] <danvet_> looking at the Jan 2020 report
[15:16:33] <daniels> *all
[15:16:54] <daniels> we can make GCE SSD smaller
[15:17:18] <danvet_> the attached one, or the PD?
[15:17:24] <danvet_> the attached one is just like 120$
[15:17:31] <danvet_> almost a rounding error
[15:17:34] <anholt> danvet_: yep, looks like I totally misremembered
[15:17:43] <daniels> danvet_: PD
[15:17:52] <danvet_> 410$ last month
[15:17:58] <anholt> the one I dug into a bit was cloud storage download
[15:18:05] <danvet_> I guess if you can shrink it somewhat helps a bit
[15:18:17] <danvet_> but if we then have outage and admin time wasted, imo not worth it
[15:18:28] <danvet_> anholt, yeah that's the big spender
[15:18:50] <danvet_> plus corresponding groth in gce network egress (I guess all the git clones and api calls)
[15:19:04] <daniels> we currently store backups on PD because backup to cloud storage broke. it should be fixed now - meaning we can drastically drop PD usage - but not had time to verify and fix
[15:19:27] <danvet_> daniels, imo not priority
[15:19:46] <daniels> migrating to another provider is absolutely ok for me. we can do it if it's a one-off thing and I can schedule it
[15:19:48] <danvet_> good to do ofc still
[15:20:05] <danvet_> daniels, yeah my idea is to essentially ask for the world from aws
[15:20:07] <daniels> but it would need to be one-off rather than happening every few months
[15:20:09] <daniels> ok
[15:20:11] <danvet_> something like 100TB/month of S3 egress
[15:20:28] <danvet_> for CI cloud storage
[15:20:33] <daniels> I can make that happen with a couple/few weeks runway
[15:20:38] <danvet_> below that with our growth I think just not worth the bother
[15:21:27] <danvet_> and probably need to make it clear that if gitlab CI for the kernel takes off, we'll need more
[15:24:24] <daniels> ofc if Google will renew our sponsorship but with more money, all the better as it saves the time
[15:24:31] <danvet_> yeah
[15:24:39] <danvet_> I also have a bunch of ideas for non-cros google contacts
[15:24:42] <daniels> and then we can use that time to seek a better long-term option
[15:25:05] <daniels> but if S3 is what we get then I can handle the transition as long as I have some notice and time to prepare
[15:28:06] <danvet_> bryce, ping on the announcement mail
[15:28:17] <danvet_> Lyude, eta for your board minutes?
[15:28:27] <danvet_> I think best if I send out the announcement right after
[15:32:48] <Lyude> danvet_: I will get them done right now
[15:35:05] <bryce> danvet_, sent it to bryce@bryceharrington.org?
[15:35:53] <Lyude> danvet_: btw you agreed to poking intel for getting the invoice process unblocked right? (just confirming for the minutes)
[15:37:08] <mdnavare> Lyude: Yes I saw Danvet's response to intel email thread poking to change the process
[15:37:55] <danvet_> bryce, pastebin link a bit up
[15:38:15] <danvet_> https://paste.debian.net/1132596/
[15:38:24] <danvet_> Lyude, yeah already done
[15:45:55] <Lyude> danvet_: also do we need to mention the xdc2020 sponsors? it came up as something we should have on the agenda but I don't think we talked about it in depth
[15:46:10] <Lyude> (last question, then I can go ahead and send this out)
[15:46:11] <danvet_> Lyude, I dropped a one-liner
[15:46:35] <danvet_> <danvet_> xdc20 sponsors: I think we're unblocked now with the private board@ thread, I'll restart that next week
[15:47:03] <danvet_> i.e. xdc20 sponsor hunt will continue next week
[15:47:22] <danvet_> maybe dont mention the private discussion that blocked it for a bit

[15:47:47] [disconnected at Thu Feb 27 15:47:47 2020]