profile
viewpoint

the8472/mldht 128

Bittorrent Mainline DHT implementation in java

the8472/fastar 96

Fast tar archiver

the8472/ffcnt 26

Fast file counting on HDDs

the8472/platter-walk 23

Rust library for HDD-aware directory traversal

the8472/reapfrog 9

Rust library for multi-file readahead / dropbehind

the8472/defrag-dirs 4

Directory index defragmentation tool for linux

the8472/greasemonkey 2

Greasemonkey is a user script manager for Firefox.

the8472/bt-dht-rs 1

Standalone BitTorrent mainline DHT node

the8472/cryptorrent-fuse 1

bittorrent storage layer encryption prototype

the8472/ipfs-jail 1

wrap ipfs daemon in firejail, run as systemd service

PR opened rust-lang/rust

Extend io::copy buffer reuse to BufReader too

previously it was only able to use BufWriter. This was due to a limitation in the BufReader generics that prevented specialization. This change works around the issue by using where Self: Read instead of where I: Read. This limits our options, e.g. we can't access BufRead methods, but it happens to work out if we rely on some implementation details.

Copying 1MiB from /dev/zero to /dev/null through a 256KiB BufReader yields following improvements

OLD:
    io::copy::tests::bench_copy_buf_reader  51.44µs/iter +/- 703.00ns
NEW:
    io::copy::tests::bench_copy_buf_reader  18.55µs/iter +/- 237.00ns

Previously this would read 256KiB into the reader but then copy 8KiB chunks to the writer through an additional intermediate buffer inside io::copy. Since those devices don't do much work most of the speedup should come from fewer syscalls and avoided memcopies.

Currently the optimization doesn't apply to things like StdinLock, but this can be addressed with an additional AsMutBufReader specialization trait.

+132 -14

0 comment

3 changed files

pr created time in 2 hours

create barnchthe8472/rust

branch : use-buf-reader-buffer

created branch time in 3 hours

issue commentrust-lang/libs-team

Implement PartialOrd<str> for String, PartialOrd<[T] for Vec<T>

When working on a sorted vec implementation, I originally implemented get() lookups using PartialOrd<SearchType>. In effect, it looked like this

Have you tried using Borrow? HashMap::get uses it.

ecton

comment created time in 15 hours

issue commentrust-lang/libs-team

`Option::is_none_or`

Imo the Option/Result/Iterator/Stream/TryStream methods are a clusterfuck. In some cases it's flat_map or and_then or then. If "these are need for consistency/to fill in quadrants in a feature matrix" is the argument then they should actually be consistent so understanding carries from one to the other.

In the current situation where each gets a bespoke name I end up not remembering the names and having to look them up quite often or relying on RA to offer the right thing. So my general stance is to not add any more unless there's a significantly stronger argument than consistency.

WaffleLapkin

comment created time in 15 hours

pull request commentrust-lang/rust

Don't require the output from libtest to be valid UTF-8

This works around the problem

Does it though? Printing non-utf8 to a windows console will just error a bit later. https://doc.rust-lang.org/std/io/fn.stdout.html

When operating in a console, the Windows implementation of this stream does not support non-UTF-8 byte sequences. Attempting to write bytes that are not valid UTF-8 will return an error.

jyn514

comment created time in a day

Pull request review commentrust-lang/rust

Port PGO/LTO/BOLT optimized build pipeline to Rust

+use crate::environment::Environment;+use crate::utils::io::move_directory;+use camino::Utf8PathBuf;+use std::io::{Cursor, Read};+use zip::ZipArchive;++pub(super) struct WindowsEnvironment {+    checkout_dir: Utf8PathBuf,+}++impl WindowsEnvironment {+    pub fn new() -> Self {+        Self { checkout_dir: std::env::current_dir().unwrap().try_into().unwrap() }+    }+}++impl Environment for WindowsEnvironment {+    fn checkout_path(&self) -> Utf8PathBuf {+        self.checkout_dir.clone()+    }++    fn downloaded_llvm_dir(&self) -> Utf8PathBuf {+        self.checkout_path().join("citools/clang-rust")+    }++    fn opt_artifacts(&self) -> Utf8PathBuf {+        self.checkout_path().join("opt-artifacts")+    }++    fn build_root(&self) -> Utf8PathBuf {+        self.checkout_path()+    }++    fn prepare_rustc_perf(&self) -> anyhow::Result<()> {+        const PERF_COMMIT: &str = "9dfaa35193154b690922347ee1141a06ec87a199";++        let url = format!("https://github.com/rust-lang/rustc-perf/archive/{PERF_COMMIT}.zip");+        let mut response = Vec::new();+        ureq::get(&url).call()?.into_reader().read_to_end(&mut response)?;

curl and pipes don't need any bash. std::process::Command::new("curl").args(...)

Kobzol

comment created time in 2 days

PullRequestReviewEvent

Pull request review commentrust-lang/rust

std: available_parallelism using native netbsd api first

 pub fn available_parallelism() -> io::Result<NonZeroUsize> {                 }             } +            #[cfg(target_os = "netbsd")]+            {+                unsafe {+                    let set = libc::_cpuset_create();+                    if !set.is_null() {+                        let mut count: usize = 0;+                        if libc::pthread_getaffinity_np(libc::pthread_self(), libc::_cpuset_size(set), set) == 0 {+                            #[cfg(target_pointer_width = "32")]+                            const MAXCPUS: u64 = 32;+                            #[cfg(target_pointer_width = "64")]+                            const MAXCPUS: u64 = 256;+                            for i in 0..MAXCPUS {+                                match libc::_cpuset_isset(i, set) {+                                    -1 => break,+                                    0 => continue,+                                    _ => count = count + 1,+                                }+                            }+                        }+                        libc::_cpuset_destroy(set);+                        if count > 0 {+                            return Ok(NonZeroUsize::new_unchecked(count));+                        }+                    }+                }+            }

The linux cgroup stuff has been extracted into its own module for example.

devnexen

comment created time in 2 days

PullRequestReviewEvent

Pull request review commentrust-lang/rust

Port PGO/LTO/BOLT optimized build pipeline to Rust

+use crate::environment::Environment;+use crate::utils::io::move_directory;+use camino::Utf8PathBuf;+use std::io::{Cursor, Read};+use zip::ZipArchive;++pub(super) struct WindowsEnvironment {+    checkout_dir: Utf8PathBuf,+}++impl WindowsEnvironment {+    pub fn new() -> Self {+        Self { checkout_dir: std::env::current_dir().unwrap().try_into().unwrap() }+    }+}++impl Environment for WindowsEnvironment {+    fn checkout_path(&self) -> Utf8PathBuf {+        self.checkout_dir.clone()+    }++    fn downloaded_llvm_dir(&self) -> Utf8PathBuf {+        self.checkout_path().join("citools/clang-rust")+    }++    fn opt_artifacts(&self) -> Utf8PathBuf {+        self.checkout_path().join("opt-artifacts")+    }++    fn build_root(&self) -> Utf8PathBuf {+        self.checkout_path()+    }++    fn prepare_rustc_perf(&self) -> anyhow::Result<()> {+        const PERF_COMMIT: &str = "9dfaa35193154b690922347ee1141a06ec87a199";++        let url = format!("https://github.com/rust-lang/rustc-perf/archive/{PERF_COMMIT}.zip");+        let mut response = Vec::new();+        ureq::get(&url).call()?.into_reader().read_to_end(&mut response)?;

You could curl to a pipe to retain that property

Kobzol

comment created time in 2 days

PullRequestReviewEvent

PR closed rust-lang/rust

Add a feature to remove the static initializer on Linux/GNU S-waiting-on-review T-libs

The explicit-init-args-with-glibc feature will disable the static initializer that is used on Linux with glibc to initialize std::env::args. The result is that the args will be empty in a cdylib on this platform.

The feature is off by default, so there is no change to any Rust users unless they build stdlib themselves and turn the feature on explicitly.

+25 -6

2 comments

2 changed files

danakj

pr closed time in 4 days

pull request commentrust-lang/rust

Add a feature to remove the static initializer on Linux/GNU

Closing this since it was rejected in the associated issue

danakj

comment created time in 4 days

issue commentrust-lang/rust

Stdlib contains a static initializer on Linux, without any way to opt out

Static initialization is known to be a problem, due to the Static Initialization Order Fiasco. They also have a measurable negative impact on startup speed, even on modern computers.

Since this is the underlying motivation: If these ever become a problem I think PRs that fix ordering issues or optimize performance of expensive initializers without removing them entirely also have a good chance of being accepted.

danakj

comment created time in 4 days

issue commentrust-lang/rust

Stdlib contains a static initializer on Linux, without any way to opt out

With my team hat on: We currently have no explicit policy. But we're resisting this change, among other reasons, because we do not want to commit to a policy where we never use static initializers. If a feature requires them on a platform we'll likely use them. If there are alternatives we'll consider them, but for example reading /proc/self/cmdline was rejected due to procfs not being available in all environments. Usually we'll try lazy or fallback approaches for problematic features, but with static initializers that doesn't seem possible since they happen at load time.


If stubbing/deleting the link section works out for you then we may consider less intrusive PRs in the future that keep std working (with that particular feature degraded) when the initializer is never called. This should already be the case, but I think we're not testing those circumstances.

danakj

comment created time in 4 days

issue commentrust-lang/rust

Stdlib contains a static initializer on Linux, without any way to opt out

In the long term we could also avoid the static initializer if a way to get argv would be added to glibc or the kernel. But considering backwards compatibility we would only be able to remove it once support for that becomes widespread.

danakj

comment created time in 4 days

issue commentrust-lang/rust

Stdlib contains a static initializer on Linux, without any way to opt out

Options we tossed around:

  • you carry the patches on your end since ultimately you would be responsible for maintaining them anyway. I assume that would still be less effort than going no_std. I believe (but have not tested) that you could apply patches to the rust-src component used by -Zbuild-std, so it wouldn't require rebuilding the entire compiler.
  • maybe some linker trickery to stub out the initializer? at least this wouldn't require any source patches
  • wait for/contribute to a build-std-on-stable future at which point we might be able to reconsider such features, perhaps at a higher level such as disabling all of std::env.

To date it has been an explicit project goal to have no static initializers.

Perhaps you could refine that policy to disallow static initializers that rely on other ones but allow those that are shallow?

danakj

comment created time in 4 days

issue commentrust-lang/libs-team

Implement `From<{&,&mut} [T; N]>` for `Vec<T>` where `T: Clone`

The Type::method syntax is somewhat poorly supported. Many things don't work because auto-deref and coercions don't happen when you use that syntax. Adding trait impls to make up for the lack of auto derefs seems like a good solution for that deficiency.

Imo it's worth checking if this can be improved on the language side, e.g. by inserting an anonymous shim function. I don't know if that would be acceptable since it would result in a different Fn type in the iterator adapter.

EFanZh

comment created time in 6 days

issue commentrust-lang/rust

Tracking issue for HashMap::drain_filter and HashSet::drain_filter (hash_drain_filter)

Currently waiting for a hashbrown release so that drain_filter can be replaced with extract_if.

DutchGhost

comment created time in 6 days

issue commentrust-lang/rust

`std::ops::Yeet` trait documentation is unhelpful (empty)

do yeet is contextual syntax, piggybacking on the reserved do keyword. Normally such documentation would go under the keyword docs. But it is an experimental placeholder so maybe it's not worth it to put much effort into documenting it until the syntax is finalized.

Sidenote: Yeet<T> is a struct, not a trait.

There appears to be a sensible one-sentence docstring explanation of my expected "(2)" (explains how Yeet relates to yeet). But that sentence doesn't appear on doc.rust-lang.org/std

But it does? Did you have the docs collapsed perchance?

image

mcclure

comment created time in 7 days

Pull request review commentrust-lang/rust

Make struct layout not depend on unsizeable tail

 fn univariant(             }             // Otherwise we just leave things alone and actually optimize the type's fields         } else {-            let max_field_align = fields.iter().map(|f| f.align().abi.bytes()).max().unwrap_or(1);-            let largest_niche_size = fields+            // To allow unsizing `&Foo<Type>` -> `&Foo<dyn Trait>`, the layout of the struct must+            // not depend on the layout of the tail.+            let max_field_align =+                fields_excluding_tail.iter().map(|f| f.align().abi.bytes()).max().unwrap_or(1);+            let largest_niche_size = fields_excluding_tail

I have pushed a change that fixes this case. Feel free to force-push over my commit if you already have your own.

lukas-code

comment created time in 7 days

PullRequestReviewEvent

push eventlukas-code/rust

The 8472

commit sha 3003d05a8b351031596eb5952746b2c80982952e

disable some layout optimizations for unsizable structs

view details

push time in 7 days

pull request commentrust-lang/rust

Make struct layout not depend on unsizeable tail

Do Container<T> layouts for Container<dyn Foo> always make the tail field zero-sized (as far as layout is concerned)? If so we should have a (debug?) assert that checks that any kind == MaybeUnsized type with a non-zero tail has the same layout as the dyn version of itself.

Layout optimizations have lots of edge-cases. Having an assert will cover more or at least make finding bugs easier.

lukas-code

comment created time in 7 days

Pull request review commentrust-lang/rust

Make struct layout not depend on unsizeable tail

 fn univariant(             }             // Otherwise we just leave things alone and actually optimize the type's fields         } else {-            let max_field_align = fields.iter().map(|f| f.align().abi.bytes()).max().unwrap_or(1);-            let largest_niche_size = fields+            // To allow unsizing `&Foo<Type>` -> `&Foo<dyn Trait>`, the layout of the struct must+            // not depend on the layout of the tail.+            let max_field_align =+                fields_excluding_tail.iter().map(|f| f.align().abi.bytes()).max().unwrap_or(1);+            let largest_niche_size = fields_excluding_tail

I found a case that still fails with this PR:

#![feature(offset_of)]

#[derive(Clone)]
struct WideptrField<T: ?Sized> {
  first: usize,
  second: usize,
  niche: NicheAtEnd,
  tail: T,
}

#[derive(Clone)]
#[repr(C)]
struct NicheAtEnd {
  arr: [u8; 7],
  b: bool,
}

type Tail = [bool; 8];

fn main() {
  let sized = Box::new(WideptrField {
    first: 0xdead,
    second: 0xbeef,
    niche: NicheAtEnd {
      arr: [0; 7],
      b: false,
    },
    tail: [false; 8] as Tail,
  });

  let unsized_value: Box<WideptrField<dyn Send>> = sized.clone();

  println!("offset in niche, sized   = {}", core::mem::offset_of!(WideptrField<Tail>, niche));
  println!("offset in niche, unsized = {}", core::mem::offset_of!(WideptrField<dyn Send>, niche));
}

prints

offset in niche, sized   = 0
offset in niche, unsized = 16
lukas-code

comment created time in 7 days

PullRequestReviewEvent

Pull request review commentrust-lang/rust

Make struct layout not depend on unsizeable tail

 fn univariant(             }             // Otherwise we just leave things alone and actually optimize the type's fields         } else {-            let max_field_align = fields.iter().map(|f| f.align().abi.bytes()).max().unwrap_or(1);-            let largest_niche_size = fields+            // To allow unsizing `&Foo<Type>` -> `&Foo<dyn Trait>`, the layout of the struct must+            // not depend on the layout of the tail.+            let max_field_align =+                fields_excluding_tail.iter().map(|f| f.align().abi.bytes()).max().unwrap_or(1);+            let largest_niche_size = fields_excluding_tail

These changes are necessary to fix it but I'm not sure if they're sufficient.

If an unsizing coercion could change which niche is considered the largest or how many bytes it spans that could alter the chosen layout in LayoutCalculator::univariant. I haven't constructed an example yet where that actually happens. But if you want to be conservative then disabling the alt-layout branch for unsized structs would be a safe choice.

lukas-code

comment created time in 7 days

PullRequestReviewEvent

issue commentrust-lang/rust

Struct tail ordered differently when unsizing

Oh, thanks for catching that. I totally didn't have unsized types in mind when doing those optimizations.

I'm now rereading the UCG on struct layout and it looks like I also invalidated its section on current behavior. But that should only require updating the UCG, not further fixes.

y21

comment created time in 7 days

more