Monero Dev Meeting Note Highlights – 2017-03-12
Logs
351 lines
fluffypony1. Greetings
fluffyponyok cool
fluffyponyagenda is at https://github.com/monero-project/meta/issues/51
fluffyponyso to start, 1. Greetings
fluffyponyhi
iDunkHi
fluffypony2. Brief review of what's been completed since the previous meeting
fluffyponyso mostly fiddly bits as we get ready for the 0.10.2.2 tag and release
fluffyponybut we'll discuss that more in 4
fluffyponyso 3. A discussion of PR 1816, & perhaps a more general discussion of C++ templating etc.
fluffyponyrelevant link: https://github.com/monero-project/monero/pull/1816
fluffyponyso part of the ongoing discussion happening there is "how much C++11 do we adopt"
fluffyponyand part of the discussion I wanted to have today is "how maintainable is C++11 for us"
moneromoooMaybe it's a bit less about C++11 and more about making things overcomplex.
fluffyponyyes
moneromoooA bit of both I guess.
fluffyponyso if I may, I thought I'd give my largely uninformed opinion, and then I'd particularly like to hear what hyc and moneromooo and tewinget and others feel
fluffyponyI feel that a lot of this ties into our long-term goals as developers and contributors to the Monero Project, and where we'd like to take things
fluffyponysome of this code complexity leads to easier management later on, but requires a steep learning curve now
hycI'm not up on the latest in C++. so when it comes to deciding what features to adopt, I'll stay out of it
fluffyponyor requires things to be a little harder to manage now, but this leads to an easier to manage set of classes / functions / etc. later on
moneromoooAre you saying that the code being more complex makes easier to maintain later ?
hycIMO though most of this stuff is complex syntax for things that have much simpler approaches already
moneromoooOK, I can buy that.
binaryFatefluffypony: "this code complexity leads to easier management later on" <– is that true for touching part of this very code too? Seems to be the opposite
fluffyponybut if our goal is just to "maintain" the current code base with a view to rewriting later, then obviously this is a silly exercise right now
fluffyponyso part of this discussion really comes down to: do we, as developers, feel that we can evolve this code to a modular, manageable set of libraries and tools? or is that a pipe dream and a parallel rewrite is a better idea?
moneromoooI think these are separate things (or easily separatable anyway).
moneromoooYou can do modular/manageable with simple code and complex code, and spaghetti/unminageable with simple and complex code.
ArticMineMy question here is: Which approach is the least likely to introduce bugs?
moneromoooSo let's do one at a time.
endogenicdo you guys know of Rich Hickey and his view/definition of simple vs complex, easy vs difficult?
moneromooofeels wary
hycI think a parallel rewrite is the way to go. least chance of breaking the current code base.
hycleast intrusive overall until we have something workable on the rewrite
fluffyponyhyc just wants to rewrite everything in C :-P
hycwell. yeah ;)
fluffyponyis a rewrite too ambitious a goal? do we want resources to focus on that and the current codebase effectively plateaus?
fluffypony(bearing in mind that there are lots of bits we can re-use, such as vast portions of the crypto / blockchainDB stuff)
moneromoooI'd rewrite bit by bit, if you want to rewrite.
hycgood question. I tink a rewrite is low priority until the curent codebase does everything we want. i.e., multisig + kovri and whatever else
pigeonsis vtnerd here? last week he had the most to say about c++ technique. The main concern was who else is comfortable with review
moneromoooLook at 0mq, which is a rewrite of just the RPC protocol. It's been a year or something.
hyctrue
binaryFateI and prolly many others have no grasp of the actual usage of epee in code base. What sort of workload are we talking about to replicate?
binaryFateball park
moneromoooShitloads :)
fluffypony0MQ's kinda a bad example, moneromooo - tewinget has had stuff on his plate that has interrupted his ability to work on Monero
moneromoooBesides you have to have both work in the same binary if you want to fork from one to the other at some point.
fluffyponybut in principle, yeah, this is a multi-year project
binaryFateok…
fluffyponymoneromooo: I wonder if we couldn't pull the project in as a submodule and just reference the common files in it that we need?
hycwell, if someone were on it full time it wouldn't be so bad
moneromoooSure, and my default view, unless proven wrong, is that it'd DoS us.
fluffyponypigeons: I've pinged him, but he's not here atm
i2polarkI agree with hyc. A rewrite in C should only become feasible once the current codebase is "feature-complete" kovri, multisig, 0mq etc etc
fluffyponyolark: the opposite might be true - the more "features" we add the harder it is to rewrite
fluffyponywhat if we approached it differently
i2polarkIf what is being worked on can be ported to C without technical debt that is good too.
fluffyponywhat if we have a new project whose focus is to build libraries out
fluffyponyso maybe it starts with a consensus library that can validate txs and blocks, and knows how to build txs and blocks
endogenicthat’s almost exactly what i was typing
fluffyponyand then the main Monero project uses that
hycsounds reasonble
fluffyponyand eventually the project becomes a collection of files that can be dropped because they're no longer needed, and just dumb interfaces that hook into libs
i2polarkA slow transition would be ideal
endogenicat that point their internal implementations can be replaced with C etc right?
ArticMineSo the rewrite is done library by library
fluffyponyit would be massively unrestricted, since everything is created from scratch and could even have (gasp) whitespace fixes
hycI was thinking about refactoring the blockchainDB stuff. it should all be hidden behind an interface class. nobody outside of blockchainDB should be including db_lmdb.h or instantiating blockchainLMDB direcftly
fluffyponyI'm not against doing it in C, and that is super portable, but by going that low-level on everything aren't we also making it harder on ourselves?
fluffyponyhyc: I agree with that
endogenicdid not intend to suggest C whatsoever
fluffyponyendogenic: too late :-P
hycright, pure C is my personal goal. I wouldn't impose that on the project.
moneromoootbh, the idea of spending shitloads of time and effort to replace with the same, in a way that hardforks every single part of the consensus seems outrageous. The only reason it doesn't is because the dangling prize (a nice, bug free C implementation which didn't fork) is not.
hycare you saying too much risk for too little reward? or possibility of no reward?
fluffyponywell let's be brutally honest here: we didn't write this code, and parts of it are well-written whilst others are simply not
fluffypony(didn't write it initially I mean)
moneromoooToo much risk for a reward that's not too easy to quantify.
endogenicwe’ll prolly be well served by enumerating exactly what interfaces we would need (i.e. solutions to what problems) in the process of judging risk-reward - cause there may be low hanging but high value fruit
fluffyponyperhaps we need to consider it this way: at what point do we NEED a massive rewrite that gets rid of all the technical debt?
hycyeah, sounds like we need an actual review.
fluffyponylike I can't imagine there still being dredges of epee in 10 year's time
hyclook at all of the existing code, draw high level boxes around pieces that make sense.
binaryFateAfter features required for decent growth of the ecosystem are there, imo
hycthen walk away, if you zoom in on the boxes again, does the layout you would reasonably expect to see match what exists.
binaryFateSo answer depends with you would trade those efforts against
binaryFatewhat
hycso the assumption is that the code we have now actually works. i.e. is solid enough to continue building on top.
fluffyponyhyc: perhaps that answer leads us to the next point in the meeting
fluffyponyso let's open that up so that it can form part of the discussion
fluffypony4. A Series of Unfortunate Events: the v0.10.2.x release failures, massive regressions, and serious issues. How do we do better in future?
fluffyponygiven how many issues we've had with 0.10.2.x, IS the code solid enough?
moneromoooThis whole thing is assuming that a rewrite would be bug free, which is a preposterous assumption.
hyctrue, most ground-up rewrites are worse than what you started with
binaryFateissues in 0.10.2 are due to new stuff not old ones, no?
moneromoooWe'd have to go find a list, but I think so, mostly.
fluffyponymoneromooo: no no, not bug free, but something with a more complete test suite, and easier to review / manage
moneromoooWriting more tests is completely independent of rewriting.
codehaloi am writing a few applications in Swift, and I had to create a lot of C bindings to the C++ code.
codehaloIt would have been so much easier if the codebase was just C…
hyc^^ for cross-language portability our public APIs should be C
fluffyponyponders
codehalothe Swift framework for LMDB has been easier to get up and running
hycnot sure how we could have tested more. the batch-txn changes had been in testing for a couple months
fluffyponyI have a suspicion that going C-only would somewhat restrict our potential contributor base
moneromoooWhy ?
hycwith several live blockchain reorgs and no problem. had no way to suspect a race condition was lurking.
binaryFateSurviving in C is possible for every C++ coder, but writing proper C for cryptocurrency software, not
fluffyponymoneromooo: I know competent C++ developers who are mediocre when working solely in C
codehaloAs hyc said, at least have a strong C api available
fluffyponysure
binaryFateYou can do crazy sneaky things in C in terms of malicious commits, look at the contests for that it's fun.
fluffyponyI agree with that
moneromoooC++ too…
binaryFateReviewers able to catch that, you'd have way less of them than in C++
binaryFatemoneromooo not if you expect writer to write proper C++, then it's obvious if it's going low-level unnecessarily
moneromoooWTF.
moneromoooWhy are half the reasons given above completely suprious ?
hycI think we need to back away from C vs C++. the codebase is primarily C++ now.
hycwe deal with what we have.
endogenici agree. porting to C for whole codebase seems an answer in search of a problem - bound to go badly - is it an API issue? or is it a performance issue? can C by applied selectively, establishing standalone libs, while enabling us to maintain the expertise, past testing, etc of the existing codebase?
fluffyponymoneromooo: spurious?
endogenicimo the biggest problem coming into monero initially for me was “how do i integrate with this?” or “what specifically do i integrate with?”
moneromoooMaybe not the right word, but… I feel like half the reasons for a choice given by people above either don't apply, or also apply to the other choice.
codehaloapi issue
moneromoooLike, you don't need C to write tests.
fluffyponymoneromooo: no I was asking if that's what you meant, you said "suprious" :)
moneromoooOr if you require "proper C++", then you should require "proper C" when comparing maintainability of both.
moneromoooOh yes, spurious.
fluffyponywell look, I'd like Monero to become a collection of well written, well documented libraries, with bindings in many languages
hycyes, fine.
endogenicspurious arguments really get a pony going
moneromooo:D
fluffyponyso the public side of the API should be C, even if the rest of the code is C++
binaryFatemoneromooo yes and I was saying there is less people able to do that in C than in C++
fluffyponylol endogenic
hycdo we really need integration at the object code level? wouldn't most integrations be via RPC?
fluffyponyhyc: no, not for libraries
moneromoooTo write underhanded code ?
endogenichyc: one problem i found was needing to use the wallet2.cpp code
fluffyponyconsider a client written in Go that just wants to use the consensus library
endogenicstill have that issue - Snipa did too
codehalo@fluffy yes. a lot of libraries is the key
hycyeah
hycactually I've started writing that in C to do the LMDB version
binaryFatemoneromooo less people to review and prevent people to write underhanded code
fluffyponyok so being able to hook into things at an object code level is a requirement
fluffyponyalso as an idea
fluffyponywhat if the "rewrite" / refactor / library effort happened in the source tree?
hychmmm. binaryFate - in that regard, C++ is a superset of C. so if you can't recognize underhanded code in C, you wouldn't have seen it in C++ either.
moneromoooLess people to review C for underhanded code, than people to review C++ for underhanded code. Assuming we require proper/nice code for both (otherwise this is a spurious argument). Maybe. It doesn't sound like a very convincing argument though.
binaryFatehyc for most part of the code a commiter is expected to work at a level of abstraction that is way above that of C
hycfluffypony wdy mean
moneromoooI'd be fine with a part-by-part rewrite as we go, as I said above.
perothere was a discussion of c++ and crypto on hn in the last week - wondering if anyone has seen it. if not, i'll drop a link after the meeting
fluffyponyhyc: instead of a parallel rewrite in a separate project, it can be done in the source tree in a separate directory, and can rewrite / move files into that directory, as long as references are adjusted accordingly
fluffyponytks pero that would be appreciated
hycstrikes me as being hard to coordinate, but ok
moneromoooI'd keep them both plugged, and fork on a height. Because they're bound to behave differently in corner cases.
moneromooo(for the consensus changes anyway, the rest can be changed at once I guess)
fluffyponymoneromooo: then it may as well be done in a separate project, unless there's another reason to keep it in the source tree?
moneromoooYes, because people won't exit monerod-c++ and restart monerod-c at the fork height all at once.
bigreddmachinedoing it in the source tree… you'd have to be working on parts of the codebase that are basically set in stone at this point, or else really keeping up with codebase, no? otherwise, wouldn't it end up like some of the other things that have been worked on in a branch that then cause merging to be hell? I don't really know what I'm talking about, but that's been what it's seemed like on a few things fro
bigreddmachinem an outside perspective at least…
hycyeah that's what I meant about tough to coordinate
moneromoooBut maybe I missed the point you brought fluffypony ?
fluffyponybigreddmachine: yes it would cause merging hell, but we merge PRs quickly enough that it shouldn't be terrible
hycwhen I've done this scale of changes on past projects, it was only after a code freeze. feature freeze.
fluffyponymoneromooo: well I was thinking about the way we did the blockchainDB
fluffyponywe first created this generic class
fluffyponyand then did the LMDB implementation for that
hycwhich is why I wouldn't bother to start here until we have kovri etc.
perofound it quicker than i thought i would - https://news.ycombinator.com/item?id=13807693
fluffyponyand THEN went back and did the in-memory implementation for that class
moneromoooThere was no fork for this, right ? People updated mostly at the same time, but with a long tail ?
fluffyponyyes
fluffyponyso the in-memory interface went from being the thing you use to having all its functions updated to exist behind blockchainDB
fluffyponywhich was effectively a refactor effort, more than anything else
moneromoooI kinda like doing it like that. Though the swapping at a fork adds security.
fluffyponyyes 100%
fluffyponyespecially for libmonero-consensus or whatever
hycok. that sounds good for mid to long term
hycwhat do we need to do now, for 10.2 ?
bigreddmachinewhy the need to swap at a fork? if the new implementation isn't 100% compatible, wouldn't it break on a resync when inspecting older-than-the-fork blocks? so fork or no, if there's a bug it's a mess?
fluffyponybigreddmachine: a corner case can come up 6 months after deployment
fluffyponybefore I get to hyc's question
moneromoooThat seems to assume a non adversarial system.
bigreddmachinefluffypony: so a fork lets you make sure it's compatible for everything to date, but if a corner case came up later, the old version wouldn't matter?
fluffyponysince vtnerd is playing around with the library stuff (conceptually), any objection to him working on this? we kinda want the library stuff for the lightweight remote wallet tool (ie. open-source MyMonero-like stuff)
fluffyponybigreddmachine: if you don't do it at a fork height then the old version might be widely deployed
bigreddmachineyep, okay makes sense, thanks
slacknanoakronmoneromooo I’m not seeing any INFO messages about ‘synchronised OK’ or ‘new block candidate’ etc. at log-level 1
moneromoooI'm fine with that, sure. I'd ask to not rewrite too much stuff at once, though, just because it DoSes reviewers.
moneromoooEspecially when the code is damn hard to read/review :/
hycheh
fluffyponymoneromooo: ok agreed
fluffyponyI'll also ask him to consider manageability / reviewability
moneromooo(by people who don't know much recent C++)
moneromoooI'm sure he can't review it quickly :P
moneromoooer, can.
i2pJJAbrall in Matlab, I suggest
fluffyponyLOL
bigreddmachineJJAbr - I can finally review something!
hycthere's an LMDB driver for Matlab, fine with me :P
moneromoooFor future things to discuss, I think it would be best if we could discuss them one at a time when they're not intertwined.
moneromoooLike, test coverage is unrelated to C or C++, rewrite or not.
hyctrue
moneromoooJust merging them makes it harder to decide each.
i2pJJAbrI'm not joking
hycI think we got sidetracked by C vs C++
fluffyponyJJAbr: when we have more libraries then that's fully feasible
fluffyponymoneromooo: that's fair
i2pJJAbrI'm somehow visionary, write it to Matlab, generate it into C/c++
fluffyponyI think we should fork Rust and create our own programming language
i2pJJAbrcheck it with Polyspace, and it works
fluffyponywith blackjack and hookers
moneromoooAbout tests, there's a set of functional tests that's long dead. It does transfers using wallet2 and daemon.
moneromoooOK, nevermind.
i2pJJAbrVisionary = see what automotive world currenty do
i2pJJAbrdoes
endogenicso confused
hycautomotive world writes millions of lines of code just to control an engine. that's not a good model.
pigeonswho is working on the ethereum solidity port? :P
endogenici am
i2pJJAbr"just to control an engine" = simple?
endogenicJJAbr: this is a dev meeting tho
endogeniccan we chat afterwards?
i2pJJAbryou don't know what youìre talking about
i2pJJAbranyways, sorry for the disturb
pigeonsmotion to move to next topic
fluffyponyok
i2pJJAbryou're talking about refactoring? do you know about what're you talking about?
endogenicJJAbr: yes
endogenicthanks
fluffypony5. Code + ticket discussion / Q & A
i2pJJAbrok
i2pJJAbrsorry to all
fluffyponyso going back to hyc's question
fluffyponywe're ready to tag 0.10.2.2
fluffyponyexcept for the race condition stuff
fluffyponyand I think there are two PRs that I need to merge
fluffyponyif we get the race condition stuff done tonight then we can tag and re-start builds, finalise actual release tomorrow
moneromoooAnd a fix for the fluffy block thing smooth found earlier today.
hycisn't that odd, fluffy blocks shouldn't have been live on mainnet yet
moneromoooSome people have been enabling it on mainnet anyway.
hycok
moneromoooWhich, in this case, is good, because we found a couple bugs without having it fully on :P
binaryFatenote fluffypony in front of me is busy IRL
fluffyponysorry about that
fluffyponyso to move on to 6. Any additional meeting items
slacknanoakronHope you get to see Bruges sometime FP - lovely city
fluffyponydoes anyone have anything else to add?
slacknanoakronNot here
fluffyponyI think we have a fairly good idea as to how we can move forward, and also what's required for the release
hyccool
fluffyponywhich brings us to 7. Confirm next meeting date/time
slacknanoakron:)
fluffyponyMarch 26th?
fluffypony2 weeks from now
moneromoooWell, we've not talked about how to make things sturdier.
fluffyponywell
fluffyponyI think the current release is obviously urgent, as it fixes bugs
moneromoooExcept when I started mentioning existing tests, but chat was spammy so got dropped.
fluffyponybut maybe the next release we need to set some testnet time frame
slacknanoakronWhat would you like to see implemented on that front moneromooo?
fluffyponylike freeze it, and then do 4 weeks on testnet
fluffyponycan always tag an older commit
moneromoooThere's also a possibility to branch when doing a release, so you can cherry-pick simple fixes.
moneromoooThat way, simple fix releases don't also come with new features that can also be buggy.
fluffyponyas long as the version's bumped
fluffyponybranching it for testing is not a bad idea
moneromoooAlso, there aren't too many people on testnet, so stuff just ends up being tested on mainnet when it's released anyway (beyond obvious stuff).
moneromooonanoakron: those tests aren't run, they've been disabled for a long while, I have no idea if they even work.
moneromoooIt was already a fairly massive effort to bring all the core tests back to life…
slacknanoakronOh right
slacknanoakronEek
moneromoooI think 0MQ RPC tests would be nice, since everything's changed there AFAICT.
binaryFateWhy not discuss this on a release-basis prior to release rather than fixing principles? Sometimes a long freeze may be superfluous, but for instance something touching fees and block sizes could use thoroughly-discussed different conditions on testnet
moneromoooSomething which might be helpful is a user simulator.
slacknanoakronUnfortunately @binaryfate the fee/block size issue, whilst it really really should be analysed and studied for a while to ensure it’s the right solution for the long run, is hindering us from growing capacity now :(
ArticMineBut one could take a two stage approach on the fee blocksize issue
ArticMineA one time increase to 250000 byte min blocksize and adding the 4x fee level
slackjollymort+1
bigreddmachinemoneromooo: user simulator, as in something just creating random txs and such? i'm happy to make that and run on my desktop. it's already connected to testnet
ArticMineThen apply jollymort's proposal from 25000 to 125000 as the second stage
huskarmoneromoo: a user simulator can be written as a system of time dependent differential equations
moneromooobigreddmachine: yes, something that actually tests the testnet. Snipa has one. It's best when run on a local testnet though, so as not to spam the main one too much :)
slacknanoakron@ArcticMine @jollymort ok :)
perodoes testnet need nodes?
ArticMineThis allows more careful review of 2) while addressing the immediate issue also
slackjollymort250kB would make a huge difference by itself; it would reduce the min.fee 4-times and reduce the size issue
slackjollymortso you could set a x4 as the default
slackjollymortand it would be the same old 0.3$
slackjollymortthen, the pools could use the min.fee
bigreddmachinei was working on a testnet faucet - it's basically done, just need to deploy - but i could easily add something that just sends out a tx or two every 5 minutes or something like that, just to create some volume
slackjollymortand normal users the x4 fee
slackjollymortso normal users never wait
slackjollymortand pool payments sit in the mempool until regular users clear out
ArticMineWithout an increase in the default fee
slacknanoakronWhat we don’t want is to be stuck in some philosophical rut like a certain other cryptocurrency…we need agreement (which I’m pretty sure we all have) that an algorithmic determination of block size is better than a human one
slackjollymortwe have that, but we made a unintended anomaly
slackjollymortexibit: https://raw.githubusercontent.com/JollyMort/monero-research/master/Monero%20Dynamic%20Block%20Size%20and%20Dynamic%20Minimum%20Fee/Images/Fig3.1-7.png
slackjollymortno wait, wrong chart
ArticMinean increase in tx size to ~13000 bytes from ~500 bytes
slacknanoakronYes, and a binomial distribution thereof
slackjollymortthis one: https://raw.githubusercontent.com/JollyMort/monero-research/master/Monero%20Dynamic%20Block%20Size%20and%20Dynamic%20Minimum%20Fee/Images/Fig3.1-6.png
slackjollymortthe spike was introduced solely by RCT
SnipaThe user simulator I have isn't particularly complex. I may see about writing a more complex one in the future if it's something we need.
slackjollymortthe green one was pre-RCT
moneromoooDunno if we need more complex for now. Yours did find bugs already :)
slackjollymortwith 250kB that spike would surely be less severe
ArticMineThat is why I propose the two stage approach
slackjollymorti can draw the charts if you want
slackjollymortmake it part of the paper, too
fluffyponycan we formally end the meeting and take meeting bot offline?
fluffyponywe're getting out of meeting stuff anyway
ArticMineSure
Sources and notes
- Meeting log: Overview and Logs for the Dev Meeting Held on 2017-03-12, getmonero.org. The official post carried the raw log and pointed readers to Hello Monero for the overview.
- Original Hello Monero URL slug recovered from the Wayback Machine index; the original page content did not survive, so this reconstruction carries the log as archived by the Monero Project.