the8472/mldht 128
Bittorrent Mainline DHT implementation in java
Fast tar archiver
Fast file counting on HDDs
Rust library for HDD-aware directory traversal
Rust library for multi-file readahead / dropbehind
Directory index defragmentation tool for linux
Greasemonkey is a user script manager for Firefox.
Standalone BitTorrent mainline DHT node
bittorrent storage layer encryption prototype
wrap ipfs daemon in firejail, run as systemd service
PR opened rust-lang/rust
previously it was only able to use BufWriter. This was due to a limitation in the BufReader generics that prevented specialization. This change works around the issue by using where Self: Read
instead of where I: Read
. This limits our options, e.g. we can't access BufRead methods, but it happens to work out if we rely on some implementation details.
Copying 1MiB from /dev/zero
to /dev/null
through a 256KiB BufReader yields following improvements
OLD:
io::copy::tests::bench_copy_buf_reader 51.44µs/iter +/- 703.00ns
NEW:
io::copy::tests::bench_copy_buf_reader 18.55µs/iter +/- 237.00ns
Previously this would read 256KiB into the reader but then copy 8KiB chunks to the writer through an additional intermediate buffer inside io::copy
. Since those devices don't do much work most of the speedup should come from fewer syscalls and avoided memcopies.
Currently the optimization doesn't apply to things like StdinLock
, but this can be addressed with an additional AsMutBufReader
specialization trait.
pr created time in 2 hours
issue commentrust-lang/libs-team
Implement PartialOrd<str> for String, PartialOrd<[T] for Vec<T>
When working on a sorted vec implementation, I originally implemented get() lookups using PartialOrd<SearchType>. In effect, it looked like this
Have you tried using Borrow? HashMap::get
uses it.
comment created time in 15 hours
issue commentrust-lang/libs-team
Imo the Option/Result/Iterator/Stream/TryStream methods are a clusterfuck. In some cases it's flat_map
or and_then
or then
. If "these are need for consistency/to fill in quadrants in a feature matrix" is the argument then they should actually be consistent so understanding carries from one to the other.
In the current situation where each gets a bespoke name I end up not remembering the names and having to look them up quite often or relying on RA to offer the right thing. So my general stance is to not add any more unless there's a significantly stronger argument than consistency.
comment created time in 15 hours
pull request commentrust-lang/rust
Don't require the output from libtest to be valid UTF-8
This works around the problem
Does it though? Printing non-utf8 to a windows console will just error a bit later. https://doc.rust-lang.org/std/io/fn.stdout.html
When operating in a console, the Windows implementation of this stream does not support non-UTF-8 byte sequences. Attempting to write bytes that are not valid UTF-8 will return an error.
comment created time in a day
Pull request review commentrust-lang/rust
Port PGO/LTO/BOLT optimized build pipeline to Rust
+use crate::environment::Environment;+use crate::utils::io::move_directory;+use camino::Utf8PathBuf;+use std::io::{Cursor, Read};+use zip::ZipArchive;++pub(super) struct WindowsEnvironment {+ checkout_dir: Utf8PathBuf,+}++impl WindowsEnvironment {+ pub fn new() -> Self {+ Self { checkout_dir: std::env::current_dir().unwrap().try_into().unwrap() }+ }+}++impl Environment for WindowsEnvironment {+ fn checkout_path(&self) -> Utf8PathBuf {+ self.checkout_dir.clone()+ }++ fn downloaded_llvm_dir(&self) -> Utf8PathBuf {+ self.checkout_path().join("citools/clang-rust")+ }++ fn opt_artifacts(&self) -> Utf8PathBuf {+ self.checkout_path().join("opt-artifacts")+ }++ fn build_root(&self) -> Utf8PathBuf {+ self.checkout_path()+ }++ fn prepare_rustc_perf(&self) -> anyhow::Result<()> {+ const PERF_COMMIT: &str = "9dfaa35193154b690922347ee1141a06ec87a199";++ let url = format!("https://github.com/rust-lang/rustc-perf/archive/{PERF_COMMIT}.zip");+ let mut response = Vec::new();+ ureq::get(&url).call()?.into_reader().read_to_end(&mut response)?;
curl and pipes don't need any bash. std::process::Command::new("curl").args(...)
comment created time in 2 days
Pull request review commentrust-lang/rust
std: available_parallelism using native netbsd api first
pub fn available_parallelism() -> io::Result<NonZeroUsize> { } } + #[cfg(target_os = "netbsd")]+ {+ unsafe {+ let set = libc::_cpuset_create();+ if !set.is_null() {+ let mut count: usize = 0;+ if libc::pthread_getaffinity_np(libc::pthread_self(), libc::_cpuset_size(set), set) == 0 {+ #[cfg(target_pointer_width = "32")]+ const MAXCPUS: u64 = 32;+ #[cfg(target_pointer_width = "64")]+ const MAXCPUS: u64 = 256;+ for i in 0..MAXCPUS {+ match libc::_cpuset_isset(i, set) {+ -1 => break,+ 0 => continue,+ _ => count = count + 1,+ }+ }+ }+ libc::_cpuset_destroy(set);+ if count > 0 {+ return Ok(NonZeroUsize::new_unchecked(count));+ }+ }+ }+ }
The linux cgroup stuff has been extracted into its own module for example.
comment created time in 2 days
Pull request review commentrust-lang/rust
Port PGO/LTO/BOLT optimized build pipeline to Rust
+use crate::environment::Environment;+use crate::utils::io::move_directory;+use camino::Utf8PathBuf;+use std::io::{Cursor, Read};+use zip::ZipArchive;++pub(super) struct WindowsEnvironment {+ checkout_dir: Utf8PathBuf,+}++impl WindowsEnvironment {+ pub fn new() -> Self {+ Self { checkout_dir: std::env::current_dir().unwrap().try_into().unwrap() }+ }+}++impl Environment for WindowsEnvironment {+ fn checkout_path(&self) -> Utf8PathBuf {+ self.checkout_dir.clone()+ }++ fn downloaded_llvm_dir(&self) -> Utf8PathBuf {+ self.checkout_path().join("citools/clang-rust")+ }++ fn opt_artifacts(&self) -> Utf8PathBuf {+ self.checkout_path().join("opt-artifacts")+ }++ fn build_root(&self) -> Utf8PathBuf {+ self.checkout_path()+ }++ fn prepare_rustc_perf(&self) -> anyhow::Result<()> {+ const PERF_COMMIT: &str = "9dfaa35193154b690922347ee1141a06ec87a199";++ let url = format!("https://github.com/rust-lang/rustc-perf/archive/{PERF_COMMIT}.zip");+ let mut response = Vec::new();+ ureq::get(&url).call()?.into_reader().read_to_end(&mut response)?;
You could curl to a pipe to retain that property
comment created time in 2 days
PR closed rust-lang/rust
The explicit-init-args-with-glibc
feature will disable the static initializer that is used on Linux with glibc to initialize std::env::args
. The result is that the args will be empty in a cdylib on this platform.
The feature is off by default, so there is no change to any Rust users unless they build stdlib themselves and turn the feature on explicitly.
pr closed time in 4 days
pull request commentrust-lang/rust
Add a feature to remove the static initializer on Linux/GNU
Closing this since it was rejected in the associated issue
comment created time in 4 days
issue commentrust-lang/rust
Stdlib contains a static initializer on Linux, without any way to opt out
Static initialization is known to be a problem, due to the Static Initialization Order Fiasco. They also have a measurable negative impact on startup speed, even on modern computers.
Since this is the underlying motivation: If these ever become a problem I think PRs that fix ordering issues or optimize performance of expensive initializers without removing them entirely also have a good chance of being accepted.
comment created time in 4 days
issue commentrust-lang/rust
Stdlib contains a static initializer on Linux, without any way to opt out
With my team hat on: We currently have no explicit policy. But we're resisting this change, among other reasons, because we do not want to commit to a policy where we never use static initializers. If a feature requires them on a platform we'll likely use them. If there are alternatives we'll consider them, but for example reading /proc/self/cmdline
was rejected due to procfs not being available in all environments.
Usually we'll try lazy or fallback approaches for problematic features, but with static initializers that doesn't seem possible since they happen at load time.
If stubbing/deleting the link section works out for you then we may consider less intrusive PRs in the future that keep std working (with that particular feature degraded) when the initializer is never called. This should already be the case, but I think we're not testing those circumstances.
comment created time in 4 days
issue commentrust-lang/rust
Stdlib contains a static initializer on Linux, without any way to opt out
In the long term we could also avoid the static initializer if a way to get argv would be added to glibc or the kernel. But considering backwards compatibility we would only be able to remove it once support for that becomes widespread.
comment created time in 4 days
issue commentrust-lang/rust
Stdlib contains a static initializer on Linux, without any way to opt out
Options we tossed around:
- you carry the patches on your end since ultimately you would be responsible for maintaining them anyway. I assume that would still be less effort than going
no_std
. I believe (but have not tested) that you could apply patches to therust-src
component used by-Zbuild-std
, so it wouldn't require rebuilding the entire compiler. - maybe some linker trickery to stub out the initializer? at least this wouldn't require any source patches
- wait for/contribute to a
build-std
-on-stable future at which point we might be able to reconsider such features, perhaps at a higher level such as disabling all ofstd::env
.
To date it has been an explicit project goal to have no static initializers.
Perhaps you could refine that policy to disallow static initializers that rely on other ones but allow those that are shallow?
comment created time in 4 days
issue commentrust-lang/libs-team
Implement `From<{&,&mut} [T; N]>` for `Vec<T>` where `T: Clone`
The Type::method
syntax is somewhat poorly supported. Many things don't work because auto-deref and coercions don't happen when you use that syntax.
Adding trait impls to make up for the lack of auto derefs seems like a good solution for that deficiency.
Imo it's worth checking if this can be improved on the language side, e.g. by inserting an anonymous shim function. I don't know if that would be acceptable since it would result in a different Fn type in the iterator adapter.
comment created time in 6 days
issue commentrust-lang/rust
Tracking issue for HashMap::drain_filter and HashSet::drain_filter (hash_drain_filter)
Currently waiting for a hashbrown release so that drain_filter
can be replaced with extract_if
.
comment created time in 6 days
issue commentrust-lang/rust
`std::ops::Yeet` trait documentation is unhelpful (empty)
do yeet
is contextual syntax, piggybacking on the reserved do keyword. Normally such documentation would go under the keyword docs. But it is an experimental placeholder so maybe it's not worth it to put much effort into documenting it until the syntax is finalized.
Sidenote: Yeet<T>
is a struct, not a trait.
There appears to be a sensible one-sentence docstring explanation of my expected "(2)" (explains how Yeet relates to yeet). But that sentence doesn't appear on doc.rust-lang.org/std
But it does? Did you have the docs collapsed perchance?
comment created time in 7 days
Pull request review commentrust-lang/rust
Make struct layout not depend on unsizeable tail
fn univariant( } // Otherwise we just leave things alone and actually optimize the type's fields } else {- let max_field_align = fields.iter().map(|f| f.align().abi.bytes()).max().unwrap_or(1);- let largest_niche_size = fields+ // To allow unsizing `&Foo<Type>` -> `&Foo<dyn Trait>`, the layout of the struct must+ // not depend on the layout of the tail.+ let max_field_align =+ fields_excluding_tail.iter().map(|f| f.align().abi.bytes()).max().unwrap_or(1);+ let largest_niche_size = fields_excluding_tail
I have pushed a change that fixes this case. Feel free to force-push over my commit if you already have your own.
comment created time in 7 days
push eventlukas-code/rust
commit sha 3003d05a8b351031596eb5952746b2c80982952e
disable some layout optimizations for unsizable structs
push time in 7 days
pull request commentrust-lang/rust
Make struct layout not depend on unsizeable tail
Do Container<T>
layouts for Container<dyn Foo>
always make the tail field zero-sized (as far as layout is concerned)? If so we should have a (debug?) assert that checks that any kind == MaybeUnsized
type with a non-zero tail has the same layout as the dyn version of itself.
Layout optimizations have lots of edge-cases. Having an assert will cover more or at least make finding bugs easier.
comment created time in 7 days
Pull request review commentrust-lang/rust
Make struct layout not depend on unsizeable tail
fn univariant( } // Otherwise we just leave things alone and actually optimize the type's fields } else {- let max_field_align = fields.iter().map(|f| f.align().abi.bytes()).max().unwrap_or(1);- let largest_niche_size = fields+ // To allow unsizing `&Foo<Type>` -> `&Foo<dyn Trait>`, the layout of the struct must+ // not depend on the layout of the tail.+ let max_field_align =+ fields_excluding_tail.iter().map(|f| f.align().abi.bytes()).max().unwrap_or(1);+ let largest_niche_size = fields_excluding_tail
I found a case that still fails with this PR:
#![feature(offset_of)]
#[derive(Clone)]
struct WideptrField<T: ?Sized> {
first: usize,
second: usize,
niche: NicheAtEnd,
tail: T,
}
#[derive(Clone)]
#[repr(C)]
struct NicheAtEnd {
arr: [u8; 7],
b: bool,
}
type Tail = [bool; 8];
fn main() {
let sized = Box::new(WideptrField {
first: 0xdead,
second: 0xbeef,
niche: NicheAtEnd {
arr: [0; 7],
b: false,
},
tail: [false; 8] as Tail,
});
let unsized_value: Box<WideptrField<dyn Send>> = sized.clone();
println!("offset in niche, sized = {}", core::mem::offset_of!(WideptrField<Tail>, niche));
println!("offset in niche, unsized = {}", core::mem::offset_of!(WideptrField<dyn Send>, niche));
}
prints
offset in niche, sized = 0
offset in niche, unsized = 16
comment created time in 7 days
Pull request review commentrust-lang/rust
Make struct layout not depend on unsizeable tail
fn univariant( } // Otherwise we just leave things alone and actually optimize the type's fields } else {- let max_field_align = fields.iter().map(|f| f.align().abi.bytes()).max().unwrap_or(1);- let largest_niche_size = fields+ // To allow unsizing `&Foo<Type>` -> `&Foo<dyn Trait>`, the layout of the struct must+ // not depend on the layout of the tail.+ let max_field_align =+ fields_excluding_tail.iter().map(|f| f.align().abi.bytes()).max().unwrap_or(1);+ let largest_niche_size = fields_excluding_tail
These changes are necessary to fix it but I'm not sure if they're sufficient.
If an unsizing coercion could change which niche is considered the largest or how many bytes it spans that could alter the chosen layout in LayoutCalculator::univariant
.
I haven't constructed an example yet where that actually happens. But if you want to be conservative then disabling the alt-layout branch for unsized structs would be a safe choice.
comment created time in 7 days
fork the8472/unsafe-code-guidelines
Forum for discussion about what unsafe code can and can't do
https://rust-lang.github.io/unsafe-code-guidelines
fork in 7 days
issue commentrust-lang/rust
Struct tail ordered differently when unsizing
Oh, thanks for catching that. I totally didn't have unsized types in mind when doing those optimizations.
I'm now rereading the UCG on struct layout and it looks like I also invalidated its section on current behavior. But that should only require updating the UCG, not further fixes.
comment created time in 7 days