Monero/Kovri Dev Meeting Note Highlights – 2016-10-30
Logs
351 lines
Jaqueeit takes quite long time to open/close wallets with this mempool size. Could the mempool be saved in wallet cache?
Jaqueeor is that a bad idea?
Jaqueedoesnt really affect closing wallets. my bad. but opening takes longer than normal i think?
realsonytest
moneromoooYou can test that easily by logging something before and after loading.
moneromoooIt's in src/cryptonote_core/tx_pool.cpp
hycis it mtg time yet?
moneromoooYes
peroromero dev conference
Jaqueeall right. the gui def takes longer time top open because of mempool. And it also waits for mempool to be synced before closing.
hycwhy is that different from CLI behavior?
Jaqueenot sure yet
Jaqueeopening seems the same
peromooo you're a whitespace nazi
slacknanoakronIs there a way that git PRs could be run through a common parser on the server-side to standardise any whitespace?
moneromoooNot really, since it'll invariably try to reformat irrelevant stuff.
moneromoooThat said, if it could run on the diffs, it might work.
slacknanoakronAnd WRT mempool size - would fluffy blocks as standard across the network (e.g. if included in HF4) make things better?
revler1082i don't think so, it just helps with not sending txs again when a block is found, the mempool size is unaffected
moneromoooSo… if the pony isn't here… anyone wants to say something dev related ?
slacknanoakronOK yes I see. But it should allow us to avoid some of the issues that bitcoin is having - if all mempools are synched and a miner wants to eat all the transactions into a single megablock, it would propagate much more quickly than without
slacknanoakronI’m a tinkerer rather than a dev :slightly_smiling_face:
revler1082yea so the bigger the blocks (assuming mempools are aligned), the greater the savings
slacknanoakronWhich would be an extra cool reason for making fluffy blocks compulsory in a hard fork
revler1082hmm, I saw monero-moo's latest comments on the fluffy-block stuff, so I was going to go through those, but I don't have much else.
revler1082if you guys would like me to change anything, etc, feel free to ask/comment
slacknanoakronMy only ‘dev issue’ is with the growing number of dead issues on github
slacknanoakronWe should probably only be at 60 or so live issues
moneromoooI think at this point, it's testing. Especially that one about the array of txes noit in the pool but not requested again.
realsonymay i as what a dead issue is?
realsonyask
slacknanoakronOne that’s been solved
realsonyty
dEBRUYNEmonero-core has a lot of those too fwiw
dEBRUYNEI can give you a list for monero-core fluffypony
hyctypically I would not close an issue until the fix is in a tagged release
hyc(at least, on other projects.)
pero^ +1
slacknanoakronOr where the codebase has evolved onwards but the original person who has raised the issue has not tested to see if it’s still active
peroor at the very least merged
slacknanoakron@hyc can I ask - completely separate issue, have you considered offering your skills to port Bitcoin Unlimited to LMDB? They’ve got $500k in the bank…
hycI hadn't heard anything about that nanoakron
slacknanoakron@hyc ask around - I’m sure their users would want to offer an even better reason to switch their node software away from core
ArticMineBitcoin unlimited will be insecure once the emission runs out
slacknanoakronAnd I think the entire ecosystem will benefit
slacknanoakronAs for dead issues - there are still ones hanging around from before 0.10.0 which either haven’t been updated by their issuers or have been addressed
hycwould definitely be a good idea to go thru and close any that havve been fixed
revler1082@moneromooo i had left a comment in the code about whether I was double verifying since all the transactions were already in the pool, anything on that?
revler1082could speed things up a bit
moneromoooI'll have a look soon. String to grep for ?
revler1082"Do I need to do this" lol
moneromooook
slacknanoakronlol
slackmalmenHow is 0qm work?
moneromoooproxmr: you'll know in ~30 min when anonimal's around for the kovri meeting.
moneromoootewinget: around ?
slacknanoakronSo the big themes I see in current development are: ZMQ, GUI, final touches to RingCT (?), changing the crypto library, changing the logging system and fluffy blocks. Any others?
slackmalmenWhat is fluffy blocks? I am missing something here
iDunkdynamic fees
revler1082@mailmen just a compact block implementation for monero
moneromoooIt's just sending txes from a block only if a peer doens't have them already.
slackmalmenHmmmm, so, instead of sending all tx the block will contain only the reference of the tx that is in another block?
slacknanoakronNo, reference to the local mempool
slackmalmenAhh
revler1082no, it just sends the tx hashes, and since most nodes have the transactions in the mempool, it just gets the full tx from there
slacknanoakronAnd if it’s not in there then you receive the missing Tx
revler1082if it's missing any, it'll ask for those
slackmalmenNice
slacknanoakronReally nice :slightly_smiling_face:
slackmalmenIt is implemented in any place already or we will be the first ones to have it?
slacknanoakronBitcoin has it
revler1082yep, multiple implementations, xthin/compact blocks
slacknanoakronbut because they’re a dirty mix of hard and soft forked rules, and have a stupid mempool policy preventing network-wide synchronisation, they won’t see as much benefit as we will
revler1082there's some way to save even more space based on some of the stuff you guys shared with me, where you send just some prefix of the tx_hash since there's unlikely to be collisions, but I think sacrificing some space to keep things simple is ok
slackmalmenHmmm, nice… And about the way the db save the blocks, I was thinking in the other day
slacknanoakron@revler1082 you should contact the original devs of XThin and ask them about those issues :slightly_smiling_face:
moneromoooI don't know how the bitcoin ones work, but given what you wrote, I'd send the index of the tx you don't have. Possibly differential encoded.
slacknanoakron@hyc is your db guy
slackmalmenInstead of the block on the the database save the full tx, can it be save the reference to other block with tx? It will use more disk io and cpu, but can save space
revler1082thanks @moneromooo that's a good one
moneromoooI'm guessing there might be a good reason why they don't do that though.
slacknanoakronWell that’s an interesting point - if you’re pulling in other transactions to build your ring signature, do you do that from the mempool or from historic data stored in the local db? I’m not sure myself
ArticMineThe idea if I understand this is to save bandwidth
revler1082@ArticMine correct
revler1082not much difference now, but when monero takes over the world, savings should increase
ArticMineYou pull from the historic data not the mempool
slacknanoakronSo not many dev issues today which is good. Just really revler and moneromooo co-reviewing fluffy block code, and maybe closing old issues?
moneromoooWe can't close old issues. We can just make lists and wait for fluffypony to have time :)
slacknanoakronTrue
revler1082What's the crypto-lib replacement? I could maybe help with that, I mean my crypto skills are le garbage, but I can change function calls, lol
slacknanoakronWould you think about the logging system in that case? The crypto currently works well enough
moneromoooThe main problem is choosing the best one. And that needs people who know them. The pony does I think.
revler1082logging or crypto? or both?
slacknanoakronMoneromooo, do you have any thoughts on where revler can best help after fluffy blocks are finished?
moneromoooWe'll need some crypto lib with a good PRNG, to replace the keccak construction we use now.
slacknanoakronhttps://github.com/monero-project/monero/issues/1271
slacknanoakronThat’s what we’re referencing here revler - already some good discussion to review there
moneromoooWherever revler1082 is comfortable, really. If the previous bit changed was network stuff, then maybe more of it ?
slacknanoakronWhat do you think the issues are with current network code?
moneromooofluffypony wanted to switch the P2P protocol to… er… something… name escapes me.
slacknanoakron;)
moneromoooZMTP.
slacknanoakronooh…sounds fancy. How about maybe helping with ZMQ??
iDunkjesus
moneromoooIt's based on epee, from the CN people.
slacknanoakron@iDunk you rang?
revler1082all sounds good, i'll look into it and help where I can
moneromoooThe P2P code is really hard to understand, though that is subjective, mayube others like it more.
iDunkyeah, I'm on the verge of really saying something
moneromoooThere are a few bugs, mainly that it leaks sockets.
revler1082if i ever get my spaces and tabs passed @moneromooo
slacknanoakronhehe
slacknanoakronWhat’s up iDunk?
moneromoooWell, it boils down to "don't change what you don't change".
iDunknanoakron: take it easy, man
moneromoooBut I agree I may be a bit too sold on clean diffs without extraneous stuff.
moneromoooThose things are easy to fix though.
revler1082lol, @moneromooo just messin, I want to kick myself when I push and see those in the diff, like mother ..
slacknanoakronRevler wants to help with network related stuff and I’m just saying what’s in the codebase that is being worked on
moneromoooAnd maybe helping with 0MQ, I guess it'll need some testing and fixing once tewinget's done with it to a point where it can be merged.
fluffyponyyes ZMTP
moneromooopony!
slacknanoakronWoo!
fluffyponyapologies
fluffyponyhad the meeting down for 7pm not 6pm
slacknanoakronDo you guys change your clocks today down there in SA?
slacknanoakronOurs went back last night. It’s now dark at 5pm. Misery.
fluffyponyno, we don't do DST
fluffyponyOH that's why
fluffyponyeverything is confusing
DaveyJonesthought sth like this :p
fluffyponyre: closing issues
fluffyponyI'm happy to close them from lists
fluffyponyand I don't think we need to wait for them to be in a tagged release per se
slacknanoakronCool
fluffyponyhave we discussed the compact blocks thing ?
fluffyponyI see there's some backlog on it
slacknanoakronBriefly
perowhat prevents dupes then?
peromight be better to tag issues with 'fixed in next release' or something and keep them open?
revler1082all the verification is the asme, it's just saving from re-sending txs?
fluffyponypero: dupe issues?
revler1082oh
peroyea
fluffyponylol
fluffyponyrevler1082: I was also confused
perosorry =/
fluffyponynp
fluffyponypero raises a good point
fluffyponyI can flag them instead
moneromoooIf people don't search the bugs list, they'll file a dupe whether it's closed or open, no ?
slacknanoakronTake https://github.com/monero-project/monero/issues/1256 as an example
slacknanoakronMay be a dupe, may not be…
moneromoooI'm fine with bugs being opened when in doubt.
slacknanoakron@moneromooo - yes they will
dEBRUYNE<fluffypony> everything is confusing <= Oh I didn't notice too
perobut they likely won't experience the issue if it's in a release
dEBRUYNEThat link I posted on reddit says 17:00 Europe time lol
fluffyponylol
fluffyponyok so re: compact blocks
revler1082got some fixes and ideas from @moneromooo that i'm gonna try
fluffyponydo we put it in a fork wrapper so that it only activates in Jan? or are we using the versionbits and making it available from now?
revler1082only big thing is do we want backwards compatibility and a little messier code, or alter existing stuff/cleaner?
slacknanoakronThe good thing about hard forks is that we don’t need backwards compatibility
moneromoooI'd rather have it in testnet first.
slacknanoakronYes
revler1082definitely
fluffyponyok let's finish the compact blocks discussion
fluffyponyPowerFlower: pleasure :)
moneromoooI'd like to know what the changes are going to be with sodium/NaCl/cryptocpp/whatever. I don't know them, so I don't have useful input.
JaqueePowerFlower: Thank you. bye
moneromoooCompact blocks: pretty good, not much to change, will need lots of testing though.
revler1082yep
fluffyponyre: backwards compatibility, I don't think we need to support an environment where compact blocks isn't supported - a node that doesn't want to use it can just never claim to already have txs
dEBRUYNErevler1082: Have you tested it on a personal private testnet?
revler1082yea, and on mainnet with the backwards stuff
dEBRUYNEoh cool
fluffyponysince we have a testnet reorg coming up we can use the opportunity to test wrapping it in a fork
moneromoooWe need to have both block types in the code at the same time though.
moneromoooSo if we do, conditional use isn't much more.
moneromoooBTW, syncing from scratch uses a different set of messages, right ?
revler1082yes i believe so
moneromoooA possible optimization would be to always include txes with too low fee, since these would have been mined by the local host.
realsonySo men i read everything, didnt understand 97% :) convinces me that XMR has the best alt dev team :)
moneromoooBut that's really not needed now.
revler1082yea, there's a few tweaks we can make, i like your send tx index instead of hash for missing
moneromoooYes, it sounds like a no brainer, so I'm suspicious that bitcoin is not doing it for non obvious reasons…
moneromooofluffypony: do you know why they use siphash hashes and not indices in the block ?
moneromooo(since you linked to that doc, I think you might know :P)
fluffyponymoneromooo: I have no idea
ontarionoob suggestion: for the next meeting how about disabling other chat unless dev when the meeting bot started?
fluffyponybtcdrak might have an idea
fluffyponyontario: if the room goes +m then only people with +v can speak, and then it becomes a closed meeting
fluffyponybetter for it to stay open, we can handle the occasional interruption
btcdrakyeah that's what to do
ontariolol sry dont know much about irc things
fluffyponybtcdrak: moneromooo had a question about compact blocks, and why it uses siphash hashes instead of indices
moneromooo(note, I know very little about bitcoin, so it could be tx indices don't make sense in bitcoin)
btcdrakmoneromooo: ask in #bitcoin-core-dev
moneromoooOK, I will, thanks.
i2panonimalmoneromooo: re: crypto, https://cryptopp.com/ has a list of all supported schemes. I know monero will have to keep supercop/ref10 impl but most others look covered (though I can't say all because I haven't yet looked at all monero crypto yet)
fluffyponymy thinking is that we can offload the sensitive stuff to TweetNaCl (ie. crypto_ops), as it's currently using SUPERCOP ref10
fluffyponythen we offload everything else to cryptocpp, including random
fluffyponyand then if we're seeing performance bottlenecking in something specific, we can use ASM implementations only in the places it's bottlenecking
moneromoooSince we use part of… libsodium ? Does this not do everything we need ?
fluffyponymoneromooo: the libsodium source isn't even complete, so we'd have to make changes anyway
fluffyponylibsodium isn't as full-featured as cryptocpp anyway
fluffyponyand not as audited as TweetNaCl
moneromoooAnything else to talk about ?
fluffyponywell
i2panonimalSo there won't be a one-size-fits-all crypto solution, eh?
fluffyponybtw moneromooo
fluffyponyhttps://en.wikipedia.org/wiki/NaCl_(software)
fluffyponyso TweetNaCl implements all of those
fluffyponyas does libsodium
fluffyponythat libsodium can do a handful of extra things is neither here nor there
moneromoooI'm curious to know why they made many versions of hte same thing.
fluffyponythe advantage is that cryptocpp can do a TON of stuff, it's FIPS 140 validates, and it uses the Boost license
moneromoooThat sounds like a recipe for pita.
fluffyponymoneromooo: of NaCl?
moneromoooYes.
fluffyponythey only made NaCl, and then they made TweetNaCl specifically to be extremely simple and auditable (so that it could be formally verified)
fluffyponyNaCl was forked and thus begat libsodium
fluffyponyso the original NaCl reference implementation has been replaced by libsodium, effectively
moneromooocrypto++ is the same as cryptopp ?
fluffyponyyes
fluffyponyalso has AES-NI support, which is great
fluffyponydoesn't have ARMv8 crypto support yet
fluffypony"Unix (OpenBSD, Linux, OS X, etc.), Win32, Win64, Android, iOS, ARM"
slacknanoakronNot may Cortex-A53 ARMv8 systems in the wild seem to have hardware crypto so far…apparently they need to pay for an extra license to enable it
slacknanoakronPine64 may do, but I’ve not bought one yet
peropine64 does
moneromoooSo if we were to switch to cryptopp, would be still keep the existing low level crypto code, or just use for new stuff ?
moneromoooThough I guess we have a readily available test suite actually :)
i2panonimalfluffypony: I believe there is limited(?) armv8 support, or do you mean specifically aes-ni?
i2panonimalgoing off of memory, would need to confirm
fluffyponyanonimal: it's not NB right now
fluffyponymoneromooo: we'd switch
fluffyponyeven if it's piecemeal
slacknanoakronSorry to ask about stuff that’s already been covered - what’s the final decision for integrating fluffy blocks? Implement as-is on testnet, then with version flags for 0.10.0, then with forking code for January, then finally abandon backwards compatible code at HF 5?
slacknanoakronSo that all nodes at HF5 will use compact blocks (of whatever improved flavour comes along in the interim)
pigeonsits not "forking code" though right?
fluffyponyno it's not
slacknanoakronSo it could in fact be made compulsory at HF 4 without backwards compatible code?
fluffyponyand I mistakenly forgot that we need to keep both block formats anyway for sync up
fluffyponyso not worth putting it in an HF wrapper
slacknanoakronUnless we checkpoint at the next HF…
moneromoooNot for 4. Seriously…
Sources and notes
- Meeting log: Overview and Logs for the Dev Meeting Held on 2016-10-30, getmonero.org. The official post carried the raw log and pointed readers to Hello Monero for the overview.
- Original Hello Monero URL slug recovered from the Wayback Machine index; the original page content did not survive, so this reconstruction carries the log as archived by the Monero Project.