Are there, are there any projects out there, other than vs PR
acm: That would like to do a project sort of brief project review this morning.
Mark Beierl: I had put something together for store purchasing is we're talking about performance tomorrow. I thought it would be good to have a, just a quick
Mark Beierl: overview of what store. Perfect. So I'll be doing that.
acm: That's great, Mark.
acm: Anybody else
Georg Kunz: Hey, this is good. I could also briefly talk about the the alrighty implementation proposal project. I mean, everybody gets I guess on this call roughly knows about this, but we're in the process of advertising. It's can just maybe say a few words about this again.
acm: That's great. Thank you. Anybody else
acm: Okay, well why don't why don't we start, why don't why don't we start out with the I know Mark, you mentioned store per weeks ago to so let's start out with store Perth.
acm: Let's go to gay or RI two and then we'll then we'll move to the vs person on slide deck and go ahead and share your screen, Mark.
Mark Beierl: Okay, one
Mark Beierl: Quick moment here. Sorry, I just gotta
Jack Morgan: And just
Jack Morgan: Doing that we're recording this session. So
Mark Beierl: And I believe I'm up and running.
acm: You are saying to trust that we were using last hour.
acm: Sorry, it's the same anti trust that we used at the start of the hour here they're under which we are discussing all of it.
acm: And the press policy.
Mark Beierl: I'm showing an anti trust policy.
acm: No.
Mark Beierl: No, you all. Hey, I'm just, I'm just reminding people that
Yes.
Mark Beierl: Sorry, I'm going. You had me going there by mistake.
Mark Beierl: So no, that's all good, it's all good. Yes, that's correct. Where we're still operating under the same anti trust rules that
Mark Beierl: Govern all Linux Foundation things so
Mark Beierl: Okay, a very quickly. What is store per I'm actually I just realized this, you know, tool to measure block storage performance in an open stack deployment.
Mark Beierl: That's actually not true anymore. We have created a container that's basically just a simple dumb container that can run under any, um,
Mark Beierl: containerized environment Cooper 90s, or Docker Swarm or whatever sort of environment. So actually start off as a tool to measure.
Mark Beierl: storage performance in any virtualized deployment, technically, it can be used for bare metal as well. But we don't get into that here right now. But the basic idea is that tests different storage.
Mark Beierl: Drivers are different storage implementations
Mark Beierl: So it's doesn't really care what the storage looks like underneath as long as you can provide it access and tell it what it's going to be profiling either a physical device or a directory on a file system.
Mark Beierl: It'll go ahead and do that measures latency, throughput and I ops when you're going to be putting in the different workloads block sizes IO queue depth loads, etc. And at the end, it's going to provide a report of performance metrics.
Mark Beierl: So why was this put together. And why is it a particular challenge.
Mark Beierl: And the issue comes down to the question of when is a desk operating at a consistent and repeatable level of performance.
Mark Beierl: There's been, you know, different attempts at profiling discs Didi has been used in different places we sort of standardized on I've standardized on F5. Oh.
Mark Beierl: As part of store. Perfect. But there's Bonnie and other measuring tools out there. But basically, we need to ensure that the results are statistically valid.
Mark Beierl: And not just a point in time snapshot that may be affected by some sort of guests warm up. For example, Sephora itself has overhead when it allocates you blocks arm and raid controllers.
Mark Beierl: They've got a caching them. So as the cash warms up, it starts to perform better on reads versus when the caches completely empty.
Mark Beierl: SSD where leveling has to kick in at some point trends reuse blocks your operating system may have a cache, there may be file system overhead that starts to kick in after a period of time. So all of these things need to be taken into account when measuring the performance. So
Mark Beierl: The storage network Industry Association or Sia came up with a test specification basically that says, okay, in order to put everyone on the same level playing field. This is how you test a disk and basically we boiled it down to very simple
Mark Beierl: Algorithm. And that is just repeatedly test over and over and over again. So keep the thing running
Mark Beierl: Take performance measurements along the way. And once the values level out and stay that way, you essentially have a set getting state of performance. So, that is what store PR puts into code for you basically
Mark Beierl: Sorry, Cedric
Mark Beierl: Has raised hand I'm not entirely sure how to I've lost my zoom window, while I'm presenting. So I don't know how to
Mark Beierl: Address that
Cédric Ollivier: So, so, Frankie, we just do a quick question.
Cédric Ollivier: To be clear here is implemented by a for you. Jackie,
Cédric Ollivier: Is it a scenario proposed by for you or is it a specific scenario is implemented in stop, have you
Mark Beierl: Implemented in store Perth. Okay.
Cédric Ollivier: Thank you. Yeah.
Mark Beierl: Fit. Oh, good to have an attempt to do a steady state.
Mark Beierl: And I don't know that they ever finished it.
Mark Beierl: So they started working on something like this. But it's, I don't know that it's necessarily implemented. The other thing is that we're collecting the metrics every minute and putting them into essentially carbon a time series database.
Mark Beierl: So that you can examine the metrics afterwards as well. So that's, that's something in above and beyond what if I oh provides so we're we're collecting the full set of metrics from fit. Oh, and putting them into the time series database.
Mark Beierl: So, oh yeah, I should mention that there's more than just the fit to run. It's all the metrics that back it up so
Mark Beierl: Yeah, here we go. Every 60 seconds the full stats are collected so FiOS executed on each target can be any VM or bare metal as long as it has SSH open because that's how we're currently communicating with it.
Mark Beierl: We also provide a workload container which is just simply alpine with SSH in it and that
Mark Beierl: You know, can be used in a containerized environment to profile whatever storage back in the container environment is providing
Mark Beierl: Provide customer workloads, which can be random sequential read right mix, etc. And every 60 seconds the full set of statistics CPU cute depth latency submission latency.
Mark Beierl: You name it. Everything that fit. Oh, provides is collected every 60 seconds and then we take a look at the history of the past statistics for latency I ops and bandwidth
Mark Beierl: And the run will terminate automatically one steady state is reach. So what
Mark Beierl: What does that look like it kind of looks like this. So the run starts out and you can see here's our read latency and blue.
Mark Beierl: As we start collecting more samples, we can see that the latency is starting to go down we collect more samples. So there in
Mark Beierl: Orange is the is 20% above average yellow is average in gray and 80% of average. So that shows that this set of statistics that we have does not meet the steady state criteria because it is
Mark Beierl: The range is too far the highest to lowest is out of range. So as we start moving forward.
Mark Beierl: We continue to do these measurements and in this case here, we can still see there are metrics that are above and below the acceptable range.
Mark Beierl: And then towards the end of the run after about 20 samples. This particular one converge and we can see from sample nine to 19 all falls into that range. So we can now report, the average as a statistically valid average
Mark Beierl: Any questions on how this steady state works.
Sridhar Rao: Hi, I have a question.
Sridhar Rao: Yeah. So for this. The CPI solution is is preferred or mandatory or how was it when we do an OpenStack we guess
Mark Beierl: Sorry, the CPU isolation.
Sridhar Rao: Yeah, the CPU opening on the
Mark Beierl: So that's, that's exactly it as far as store perfects concerned, it doesn't care and it is purely there to act as a measurement tool so store perfect does nothing. As far as setting up the environment goats. This is meant to be.
Mark Beierl: You know, say for example the CMT t where somebody has provided a
Mark Beierl: OpenStack or a containerized environment.
Mark Beierl: Star perfect not going to go in there and modify any of that it is simply going to report on what exists and how it performs as it is.
Sridhar Rao: I agree, but when it comes to the steady state measurements right so far in this measurement is a prefer to have the CPU pinning done or if it's not like that.
Mark Beierl: It really depends on what it is that it's being
Mark Beierl: emulated, if you will, by the VM. If you are looking at profiling a workload of a VM that would have seen
Mark Beierl: Then yes you would be, go ahead and create your VM for the profiling with all the pinning done
Mark Beierl: You know, is
Sridhar Rao: Yeah yeah yeah
Mark Beierl: It's a tough question to answer. And basically we leave it up to, well, what is it that you are trying to test. So, and that is
Mark Beierl: One of the reasons why you know we've we've waited until
Mark Beierl: You know tomorrow, I guess, to really start diving into some of the performance topics on CMT t because there are a lot of questions that we're going to need to ask before we even begin to understand what it is that we want to mention
Mark Beierl: So I just got a quick wrap up here, what's next.
Mark Beierl: I'm hoping to be able to that they fit. Oh, client server model has improved and and come far enough along that will be able to get rid of SSH and just have F5 to communicate back directly with store perfect not need to use SSH as a communication with them, but that
Mark Beierl: comes with its own challenges specifically when it comes to, you know, bare metal profiling.
Mark Beierl: Because somebody would need to start up the fit. Oh, there. And we also would need to, you know, building packages specific version of fit. Oh, for the VM as opposed to being able to copy fit into the VM, which is what we're doing right now.
Mark Beierl: The big questions that are coming up what performance slash workload makes sense for see MTT
Mark Beierl: A while back two years ago I wrote a little bit of a blog post on you know the challenges for measuring guess performance specifically when it comes to a rather large safe environment.
Mark Beierl: You know, say you've got three storage machines with 384 gig of RAM each. So all of that, if these are dedicated storage machines. All of that is cash that RAM. Well, almost all of it. So that means for any sort of workload.
Mark Beierl: One rule of thumb that we've had is five times the amount of cash is the size of your test performance run. So, you know,
Mark Beierl: Is it is 700 Meg sorry 700 gigabyte times five is 3.5 terabytes is approximately the size of data that you should be reading and writing in order to get a proper profile measurement of your safe environment. In that case, so
Mark Beierl: So what to report on how these are things that I look forward to tomorrow and I probably took more time than I actually meant to with this. My apologies. Any further questions.
acm: I see Cedric has his hand up again. God said
Cédric Ollivier: Yes, I have a question, but yes.
Cédric Ollivier: I didn't know and before. So regarding the client server model. I think it's not mandatory Eve and it is case you could run a for you as a classical binary
Cédric Ollivier: Without implementing client and server. And also, if I was if I always proposed in the shake your image used by from test. So feel free to use it if it works.
Mark Beierl: Okay.
That's great. So
Okay.
Cédric Ollivier: So regarding so cash. I was not so clear because instead towards you can disability and even in Canada, where you can disable cash.
Mark Beierl: If we're talking about Seth as a back end. As an example, and you have three storage nodes for Seth and each of those nodes has got, you know, a couple hundred gigabytes of memory, you're not going to be disabling that
Mark Beierl: That kind of
Cédric Ollivier: gives you such yes but in case if you're gonna do for your honors plans of EMT it normally you but I see your point. But what I wanted to highlight that Zack cases where you can simply disabled cash on co parenting system number
Mark Beierl: Yes and no, um, you know, just because you're doing direct dial from fit. Oh, doesn't mean that the hypervisor isn't going to be intelligent about its cash. So you're not necessarily going into the hypervisor and disabling all of that.
Mark Beierl: I see. Yeah. So there's, there are multiple layers that need to be considered. And that's where it gets tricky.
acm: So, so I'm gonna, I'm going to stop this one there. Mark, and thank you very much for your presentation. Thank you. Cedric and Sridhar for your questions, good discussion and
acm: Let's hear from Georg quickly on the RI dash to Cooper 90s implementation and standard deviation, please go ahead and Georg
Georg Kunz: Right. Thanks. I guess I can make up some time is that is basically just a teaser or announcement.
Georg Kunz: I hope you can see my screen. Yeah.
Georg Kunz: Right. So as most of you are aware, there is a strong demand.
Georg Kunz: Both from CMT and also the MVP side of things, for having a container based specifically company that is based reference architecture. Well, no, sorry.
Georg Kunz: Of course reference architecture, but also reference implementation. Right. And that would serve as the baseline for instance RC two and the MVP program.
Georg Kunz: Basically, the three most important use cases for such a reference implementation is we would use it for verifying design decisions and assumptions that we've taken or assume basically while writing about ray tube or designing over P judo
Georg Kunz: Can we use for validating the select the test cases that we'd like to make part of the reference conformance, for instance, and
Georg Kunz: Third important use case is that reference implementation should be used by vendors to really test validate and then develops enough implementations against right that's, that's one of the fundamental
Georg Kunz: Goals also have seen TT and there is of course now ready to work stream ongoing and that's good and fine.
Georg Kunz: But it has been focusing on those backs parts on the documentation side of things and get up and then needs to be like a complimentary practical implementation.
Georg Kunz: Alongside and well I think open V should facilitate that development job by having a dedicated project for that in place so that become the home of the implementation implementation specific discussions and
Georg Kunz: All the, the obvious non very rocket science, science things like repose CI CD resources lab resources and stuff like that.
Georg Kunz: And the proposed approach as of today is to make use of previously proposed cabinet is bare metal reference architecture mainly developed by until
Georg Kunz: So they're behind the link. I guess I can also upload the slides at some point I should actually behind the link, you'll find that proposal more details on that.
Georg Kunz: And for some time already some months and weeks. Basically, we've been looking like on the side into using that in an open V lab. And that actually provide or requires a two phase approach because
Georg Kunz: Both this company that is deployed as well as one other obvious examples in of test that both basically are answerable playbooks that didn't work or bananas, but they both expect a pre provision operating system to be in place so it since we would like to
Georg Kunz: Do an end to end the typical open up the way we need to do we need to have an air hostess provisioning phase first
Georg Kunz: Then we put communities and tow that and then yeah, obviously, we would run, select the test cases and test tools they that's missing here. But that's not really part of the era to work.
Georg Kunz: So that's what we going to, or at least intended to do right now, as I said, it's mainly prototyping. We're like in the process of creating the project. Hence this announcement here.
Georg Kunz: We will have a deep dive session on Thursday, everybody's highly welcome to join.
Georg Kunz: To do well following things like review the current state the scope. The, the approach the proposal.
Georg Kunz: Michael pettersen from Intel has done gap analysis like what's the difference between the RA RA to requirements and implementation.
Georg Kunz: That we currently have. And then out of this. Obviously, we need to define tech steps, besides just creating the project, but also how to close those gaps and on the reference implementation slide. Thanks.
Georg Kunz: Yeah, so since this is all very much still in process, it's your chance to shape this one way or the other. So please join the session on Thursday.
Georg Kunz: Provide your insights and feedback to the discussion and then yeah, looking forward to that. And then let's see where we can
Georg Kunz: drive this thing over time.
acm: That's great. Okay, or. Thanks very much.
acm: We don't, we don't have time for questions right now, but we will have time for questions later in the week and also at the TASC meeting tomorrow I plan to have this on the agenda and we may be able to make some decisions there so oh
acm: Okay if folks feel comfortable will will do that i thanks very much for contributing today.
Georg Kunz: Welcome
acm: Okay, so, so the the final project proposal here. Let's see. Oh, street ours. Got a screen up. That's great.
acm: And the anti project and type project. Notice
acm: The antitrust notices is there. So that's the same as it was at the top of the hour and Sridhar. Go ahead. But I think I'll probably try to cover the recruiting part of this quickly and then the two of us can sort of work through the rest
acm: streetcars. The PDL of vs per project. And I've put myself down as expert work or it's it's a little bit more than that.
acm: So let's go ahead. Yeah.
Sridhar Rao: So again, because I was thinking the same and I'm the best policy.
Okay, alright.
acm: So,
acm: Let's see, is this is this now broken down into
Sridhar Rao: Yeah, yeah. Point point point point
acm: All right. Okay. So you know we we've we've made a lot of progress with interns. But one of the first things we noticed was we had lots of app applicants. Sometimes as many as 10 or 12 applicants for one position.
acm: And we ended up turning away, you know, five or six highly qualified people and doing all these interviews has been challenging
acm: And many completely satisfied satisfactory student developers were lost and that's sad when when the successful applicant then fails for some unforeseen reasons and we don't get the we don't get a student or intern task done so three dar recognize this and and
acm: We bless September we put together a proposal that op NFP would recognize
acm: student volunteers, and that's what we call them. They're unpaid, and they would work for on the basis of receiving a certificate or a letter of recognition alone and
acm: We have had one successful student volunteer work through the program, the TASC approved his completion and letter in early 2020
acm: And and his, his advisor also joined the project at many meetings and then contributed
acm: Vic. And so that was great. So, um, we, we managed to onboard new developers to the vs per project another student advisor pair has joined a part in his advisor have also stayed on parts became a an intern on a different project. And that's great too.
acm: We also have two little elephant internships in progress. And as I said before, we, we had, we had to turn away a few people. But those three of them have become student volunteers and we're working out what tasks, they'll take
acm: So here's what we've what we've learned a successful recruiting for student volunteers for involves some traveling in the academic circles. That's a
acm: Thing thing to keep in mind and having a plan and tasks for those who don't win the internships is a good thing.
acm: We need to offer tasks and projects that has some academic interest, but we need to make it clear that also the project needs help in running, you know, keeping running maintenance.
acm: And we need folks to help out with that. A key point here though street are made yesterday.
acm: Quality is not compromised sometimes delays are encouraged because we're bringing new people on they may take longer, but eventually it will get done and done with high quality and
acm: Many contributions are not recorded in the piano fi repose. It's often easier to work and GitHub because of access issues because you know stuff is up and down. That's just a reality of our lives. Next slide.
Sridhar Rao: Yeah.
acm: Go ahead.
acm: Okay.
Sridhar Rao: So thanks. Thanks. So currently we spoke. We are on 10 members who are active of each year from the software industry.
Sridhar Rao: The remaining seven are from academia. So this is the current state of this project. But, uh, we are we are going on and good going good thing. So if you're if you're aware of this per project. So initially we had a set of
Sridhar Rao: Better which is which benchmarking with different virtual switches and different VN F's we look, we interview focus mostly on the VPN.
Sridhar Rao: With the noisy neighbors and other use cases with various deployment apologies like PvP P2P and all those things.
Sridhar Rao: Now we have three main categories of words that are ongoing within vs perv. One is a covenant is networking testing and benchmarking. So we are mainly focusing on the duplicate based workloads.
Sridhar Rao: And using started with the user space CNI with the multi us and we are starting with three things. One is obviously pretty K Sarah up and up.
Sridhar Rao: And he wanted to be fully, fully automated say when I say fully automated starting from the cluster deployment to the
Sridhar Rao: CNA installations configurations, the pod deployment. The test drive traffic generation collection of the reserves, all these things. Okay.
Sridhar Rao: So this is, this is one category category that is going on. That's where we have another student volunteer and faculty from Italy. Italy, the University of Pisa. So they're working, they're working with us on this project and the next part is the way initially when we started right
Sridhar Rao: It was independent of OpenStack
Sridhar Rao: So we, we never had this open stacks of so called support. Of course we could have used it as a traffic jam mode and
Sridhar Rao: Now we want to bring in complete OpenStack support in mainly considering the CMT use case thing where we want to deploy these test VM. So it could be traffic generator. It could be load generator
Sridhar Rao: And perform the setup of those SPS and run the test and collected a sense
Sridhar Rao: Okay, that we added, we are trying to reuse some of the existing works. Of course, we don't have to do it from beginning projects like shaker gave a lot of inputs from us in doing this kind of auto deployment and running the test.
Sridhar Rao: And the third or third aspect is where we have a set of interests working on enhanced LMA that what we want to call logging monitoring alerting and analytics considering both the bare metal and the Open Stack environment.
Sridhar Rao: Okay, so there we have started this work on vs of content working benchmarking mostly focusing on the words that wasn't existed till now. So we have the dinner thorough
Sridhar Rao: Literature study kind of a thing. We have unless what all the content working performance studies that already exists. Where is the gap and trying to float the sketch. We are focusing more of the telco use cases where the the containerized
Sridhar Rao: Network functions will have more multiple interfaces and typically they'll be data plane optimize such as DP DK based data planes.
Sridhar Rao: So we are considering such a scenario and trying to perform the benchmarking. So we have a detailed working session on Wednesday. So Wednesday at 12am
Sridhar Rao: UTC so please. We are we are rolling in, in a, like a hack fest mode, we want to showcase step by step of each and every CNI how it's automatically deployed and run
Sridhar Rao: And we are we are stuck. Also, we want to see. So that does a user experts on VP who can help us in the progressing from where we are stuck.
Sridhar Rao: The VP is one of the case where we are stuck in terms of having both the mammoth and see if we can figure it out. So if you if you're interested, please attend this session, and it'll be like a hack fest can have a session where we want your inputs we want your solutions also
Sridhar Rao: Okay. And this is the in terms of OpenStack support in VS Code where we want to showcase the role of vs perv both in terms of bare metal benchmarking. We just before the VM deployment and also the post deployment. The performance testing thing and we we started working on this existing
Sridhar Rao: Currently, because of some of the things that are lacking within the response if I have to freeze that now. For now, once I complete this
Sridhar Rao: The activity, then we will be fully ready to run with existing currently the only more we are running with existing is the the bash feature mode we as we want to fully better integrate. So we are working along with the their project called
Sridhar Rao: Duplicate proxy project rapid thing. So, so that will be uniformly integrate with the existing going
Sridhar Rao: Yeah, so this is the real quick, somebody off of the enhanced
Sridhar Rao: Logging monitoring and analytics that we have. We are using these kind of set of software for both the bare metal and OpenStack environments. We have a unified alerting in terms of from both from previous alert manager and elastic search
Sridhar Rao: Or key bonus alerts. So we want to unify these alerts and handle it unified way and give a better analysis to the user.
Sridhar Rao: And when it comes to analysis. We are sick for currently focusing on these three things trend analysis last classification and the causation analysis I think using the Jupiter notebooks.
Sridhar Rao: Yeah. So this was a quick summary of the current ongoing works on whisper.
Sridhar Rao: As you know, the things have changed slightly that we are trying to adapt and trying to focus on the interest of the contributors, as we have no
acm: That's great, thanks. Sridhar for finishing that up. What would you put it back on the on the twister slide.
Jack Morgan: And we also have a question.
Sridhar Rao: Yeah, yeah. So Stephen
acm: Ah, OK. We shredders got his hand up. I just wanted to comment that
acm: Street are made this slide that looked
acm: Incredibly like the twister game without ever knowing about the game twister again.
acm: So, and if you're unfamiliar with it, just Google the twister game and you'll, you'll see what I mean.
acm: Pretty close likeness. Um, yeah. Cedric guy you've got your hand up. Please go ahead.
Cédric Ollivier: Yes, a super comment.
Cédric Ollivier: Just a lot of benchmark Benchmarking Tool. It's been included in a cross testing sidebar last week. So in the Kibana and you may be interested in the latest changes.
Cédric Ollivier: Most of the cross testing site but out to deploy as a car in a few comments, so it's that from this functional part adding as it STB this repository, etc. But now we have a few
Cédric Ollivier: More benchmarking related components, such as in the document.
Sridhar Rao: So I would
Cédric Ollivier: I would send users of change.
Sridhar Rao: So now we are not focusing on the these tools at all where once we have the data we are focusing more on analysis part
Sridhar Rao: And the one is the alert is we are not ready for example, Alex, how to configure the lead is not important. Once we get the alerts. How do we do sinking. So we are focusing more on the next level of the analysis part
Is
Cédric Ollivier: This I see my printer just that you can leverage on additional components regarding the overall cost distinguish. Thank you.
Sridhar Rao: Yes. Yes. Yeah. And I when when they land that processing. I could see the whole whole orchestra comes up nicely and we make use of all of these things distance even that the menial the artifacts setting. Once we push all those artifacts their record or interesting honest.
Cédric Ollivier: Yes. And now it's so proposed.
Cédric Ollivier: By those gateway instead of me know if you prefer. So the default is still manual because mineo is is lightweight and works very well, but you could use some of those gateways you prefer instead both works well.
Cédric Ollivier: For last week as well.
Sridhar Rao: Okay, okay. Oh, I didn't know that. Because one of the other internet started focusing on the media because they're accessing had menial
Cédric Ollivier: Yeah, I want to select menial, but if you prefer at those get to it was mostly because of a discussion internally internal to own up
Cédric Ollivier: All we can
Cédric Ollivier: Interact with next is to force the logo which is bad by design, but
Cédric Ollivier: I did read this gateway in across this thing. Sorry, just to demonstrate two weeks.
Sridhar Rao: But me and Luca waiting for your documentation on better integration with existing
Cédric Ollivier: Yes, yes. On top of my agent. Yes, thank you, thank you for the remainder. Sure.
Sridhar Rao: Yeah, Jackson, you had a question.
Jack Morgan: Oh no, I was just mentioning that there was a question that said
Sridhar Rao: Okay, so, yep. Thank you.
acm: Great. Any other questions.
acm: Okay. Well, we hope to see you at the the deep dive session for this, the true actress session.
acm: You know, that means you're going to see terminal screens and and real work going on on. That's the kind of thing we like to do in vs person OPM fee in general. So we'll gather together around the campfire and help each other, solve problems. That's, that's what we do best.
acm: And we're about to we're about to start the break, and that means we all get a half an hour here for our