|
|
|
Posted
about 1 year
ago
by
Paweł Hajdan, Jr. (phajdan.jr)
I'm only beginning my experiments with SELinux, and neverallow (which is basically an assertion that prevents inserting certain allow rules) seemed a bit weird to me. While experimenting with some local policies though, after an update
... [More]
(selinux-base-policy and other sec-policy packages) my local policy triggered a neverallow rule about sys_module capability being unnecessarily granted.
In fact, re-compiling the local policy and loading the new version made the error disappear. Now this is indeed useful, because binary policy files are arguably harder to inspect, and if they get out of sync with the base policy, it's easy to introduce errors like in this case.
Another conclusion is that learning takes time: it was the update that triggered this situation. I wonder what else awaits me in the SELinux land. ;-) [Less]
|
|
Posted
about 1 year
ago
by
Arun Raghavan (ford_prefect)
Derek Foreman has finally written up a nice blog post about his Androgenizer tool, which we’ve used for porting PulseAudio, GStreamer, Wayland, Telepathy and most of their dependencies to Android.
If you’ve got an autotools-based project that you’d like to build on Android, whether on the NDK or system-wide this is really useful.
|
|
Posted
about 1 year
ago
by
Diego E. Pettenò (flameeyes)
It’s been a while since I last wrote about parallel building. This has only to do with the fact that the tinderbox hasn’t been running for a long time (I’m almost set up with the new one!), and not with the many people who complained to me that
... [More]
spending time in getting parallel build systems to work is a waste of time.
This argument has been helped by the presence of a --jobs option to Portage, with them insisting that the future will have Portage building packages in parallel, so that the whole process will take less time, rather than shortening the single build time. I said before that I didn’t feel like it was going to help much, and now I definitely have some first hand experience to tell you that it doesn’t help at all.
The new tinderbox is a 32-way system; it has two 16-core CPUs, and enough RAM for each of them; you can easily build with 64 process at once, but I’m actually trying to push it further by using the unbound -j option (this is not proper, I know, but still). While this works nicely, we still have too many packages that force serial-building due to broken build systems; and a few that break in these conditions that would very rarely break on systems with just four or eight cores, such as lynx .
I then tried, during the first two rebuilds of world (one to set my choices in USE flags and packages, the other to build it hardened), running with five jobs in parallel… between the issue of the huge system set (yes that’s 4.24 years old article), and the fact that it’s much more likely to have many packages depending on one, rather than one depending on many, this still does not saturate the CPUs, if you’re still building serially.
Honestly seeing such a monstrous system take as much as my laptop, which is 1/4 in cores and 1/4 in RAM, to build the basic system was a bit… appalling.
The huge trouble seem to be for packages that don’t use make, but that could, under certain circumstances, be able to perform parallel building. The main problem with that is that we still don’t have a variable that tells us exactly how many build jobs we have to start, instead relying on the MAKEOPTS variable. Some ebuilds actually try to parse it to extract the number of jobs, but that would fail with configurations such as mine. I guess I should propose that addition for the next EAPI version… then we might actually be able to make use of it in the Ruby eclasses to run tests in parallel, which would make testing so much faster.
Speaking about parallel testing, the next automake major release (1.13 — 1.12 was released but it’s not in tree yet, as far as I can tell) will execute tests in parallel by default; this was optional starting 1.11 and now it’s going to be the default (you can still opt-out of course). That’s going to be very nice, but we’ll also have to change our src_test defaults, which still uses emake -j1 which forces serialisation.
Speaking about which, even if your package does not support parallel testing, you should use parallel make, at least with automake, to call make check; the reason is that the check target should also build the tests’ utilities and units, and the build can be sped up a lot by building them in parallel, especially for test frameworks that rely on a number of small units instead of one big executable.
Thankfully, for the day there are two more packages fixed to build in parallel: Lynx (which goes down from 110 to 46 seconds to build!) and Avahi (which I fixed so that it will install in parallel fine). [Less]
|
|
Posted
about 1 year
ago
by
Arun Raghavan (ford_prefect)
Some of you might’ve noticed that there has been a bunch of work happening here at Collabora on making cool open source technologies such as GStreamer, Telepathy, Wayland and of course, PulseAudio available on Android. Since my last blog
... [More]
post on this subject, I got some time to start looking at replacing AudioFlinger (recap: that’s Android’s native audio subsystem) with PulseAudio (recap: that’s the awesome Linux audio subsystem). This work can be broken up into 3 parts: playback, capture, and policy. The roles of playback and capture are obvious. For those who aren’t aware of system internals, the policy bits take care of audio routing, volumes, and other such things. For example, audio should play out of your headphones if they’re plugged in, off Bluetooth if you’ve got a headset paired, or the speakers if nothing’s plugged in. Also, depending on the device, the output volume might change based on the current output path.
I started by looking at solving the playback problem first. I’ve got the first 80% of this done (as we all know, the second 80% takes at least as long ;) ). This is done by replacing the native AudioTrack playback API with a simple wrapper that translates into the libpulse PulseAudio client API. There’s bits of the API that seem to be rarely used(loops and markers, primarily), and I’ve not gotten around to those yet. Basic playback works quite well, and here’s a video showing this. (Note: this and the next video will be served with yummy HTML5 goodness if you enabled the YouTube HTML5 beta).
(if the video doesn’t appear, you can watch it on YouTube)
Users of PulseAudio might have spotted that this now frees us up to do some fairly nifty things. One such thing is getting remote playback for free. For a long time now, there has been support for streaming audio between devices running PulseAudio. I wrote up a quick app to show this working on the Galaxy Nexus as well. Again, seeing this working is a lot more impressive than me describing it here, so here’s another video:
(if the video doesn’t appear, you can watch it on YouTube)
This is all clearly work in progress, but you can find the code for the AudioTrack wrapper as a patch for now. This will be a properly integrated tree that you can just pull and easily integrate into your Android build when it’s done. The PA Output Switcher app code is also available in a git repository.
I’m hoping to be able to continue hacking on the capture and policy bits. The latter, especially, promises to be involved, since there isn’t always a 1:1 mapping between AudioFlinger and PulseAudio concepts. Nothing insurmountable, though. :) Watch this space for more updates as I wade through the next bit. [Less]
|
|
Posted
about 1 year
ago
by
Diego E. Pettenò (flameeyes)
For those who wonder why Excelsior is now built for a week and it’s still not running in full capacity, the reason is relatively simple: I’m still trying to figure out how to handle network security so that I can give other developers access to
... [More]
it, and not risk that we have a security breach in other places.
The first problem is that the current setup I need to use and the final one will be very different: right now the system is connected to the internal network of my employer, while in production it will be DMZ’d, with a public IP available straight to it. The latter setup will let me set up an IPv6 tunnel, which means all the internal containers will be available by themselves, while the current one prevents me to set it up at all.
Previously this was much easier to deal with simply because I had control over an external firewall myself, which took care of most of the filtering, which combined with LXC networking options made it decently easy to deal with. Unfortunately this time I have to do without this.
One of the things that might not be obvious from the post above, nor from the documentation, is that the three setups I have drawn are the ones that require the least amount of configuration on the host-side: if you use the standard bridge, you just need to start net.br0 (or equivalent), and LXC will connect the host-side virtual Ethernet pair to the bridge by itself. If you’re using the MACVLAN modes instead, the containers will piggy-back the interface of the host, and then rely on the external router/firewall/switch to deal with the configuration — by the way I’m glad that man lxc.conf now actually says that you’re supposed to have a reflective switch to use those modes. You can probably guess it’s not that great an idea if you expect lots of internal communication.
What I’m going to do for now is setting up a system that is defended as much as possible by depth, with iptables carrying out enough filtering that I should be realistically safe. Unfortunately just iptables is not enough and what you need is iptables and ebtables (for Ethernet Bridging filtering), to make sure that the containers’ don’t dupe your IPs or something.
The idea is that one IPv6 is enabled, you can jump straight into the tinderboxes, which is required to control it, but until then, the host system acts as a jump host through a custom scponly setup, which only allows forwarding, as ProxyCommand, port 22 of others boxes within that same system.
I’d like to show more documentation of what I’m trying to do, and what I achieved already, but to do so, I’m afraid I’ll be needing some more … visual examples, as it’s very hard to explain it in words, while it should be much more clearer with a series of drawings. I guess I’ll start working on them soonish, maybe if Luca can package Synfig for me…
For now, this is enough, I guess. [Less]
|
|
Posted
about 1 year
ago
by
Sven Vermeulen (swift)
Today I’ve stabilized the sec-policy/selinux-* packages that provide the 20120215 “series” of SELinux policies. Together with the stabilization, the more recent userspace tools (like the policycoreutils as well as libraries like libsemanage and
... [More]
libselinux) have been pushed out as well. I will be dropping the older policies and userspace tools soon (as they are now deprecated). The documentation has been updated to reflect this too.
Some of the enhancements include
support for permissive domains (allowing users to mark one specific SELinux domain, such as mplayer_t, as permissive (even though the rest of the system is running in enforcing mode)
support for file context translations, so we can now say “/usr/lib64 (and below) should have the same contexts as /usr/lib”
support for role attributes, which means for policy developers, we now have similar freedom as with type attributes
support for named file transitions, so a policy rule can say that domain A, if creating a file in a directory labeled B, then that specific file should have label C. Same for directories, btw.
Although some of these enhancements were available as features individually, the policies we had were not aligned with it – and now, that has changed ;-) [Less]
|
|
Posted
about 1 year
ago
by
Diego E. Pettenò (flameeyes)
Okay so Excelsior is here, and it’s installed, and here starts the new list of troubles, which seems to start, as usual, with my favourite system ever: LXC which is the basis the Tinderbox work upon. The first problem is not strictly tied
... [More]
to LXC, but one of the dependencies required: the lynx browser fails to build in parallel if there are enough cores available (which there certainly are here!). This bug should be relatively easy to fix but I haven’t had time to look into it just yet.
A second issue came up due to the way I was proceeding to do the install, outside of office hours, and is that the ebuild is using the kernel sources, I think to identify which capabilities are available on the system. This should be fixed as well, so that it checks the capabilities on the installed linux-headers instead of the sources, which might not be the same.
The third issue is funny: Excelsior is going to use an hardened kernel. The reason is relatively simple to understand: it’s going to run a lot of code of unknown origins, it’ll allow other people in, one wants to be as as possible… unfortunately it seems like this is not, by default, a good configuration to use with LXC.
In particular, grsecurity is designed by default to limit what you can do within a chroot, by applying a long list of restrictions. This is fine, if not for the fact that LXC also chroots to start its own set up process. I’m now fixing the ebuild to warn about those options that you have to (or might) want to disable in your GrSec setup to use LXC.
Interestingly, it’s not a good idea to disable all of them, since a few are actually very good if you want to use LXC, such as the mknod restriction, which is very good in particular if you want to make sure that only a subset of the devices are accessible (even when counting in the allowed/non-allowed access of the devices cgroup).
In particular, these have to be disabled:
Deny mounts (CONFIG_GRKERNSEC_CHROOT_MOUNT)
Deny pivot_root in chroot (CONFIG_GRKERNSEC_CHROOT_PIVOT)
Capability restrictions (CONFIG_GRKERNSEC_CHROOT_CAPS)
while the double-chroot would be counter-synergistic as it would disallow services within the container to further chroot to allow a defense-in-depth approach.
Then there is another issue. Before starting to set up the actual tinderbox, I wanted to prepare another container, which is the one I’ll be using for my own purposes, including bumping of Ruby packages and stuff like that. Since the system is supposed to stay very locked down, this time I want to mount the block device straight into the container, which is a supported configuration…. but it turns out that the configuration parser, trying to workaround old issues (yes that’s a one and a half years’ old post) will ignore any mount request that doesn’t have the destination rootfs prefixed.
Unfortunately when you mount a block device, it means that you’ll end up with something along the lines of /dev/sdb1/usr/portage. This also collides with the documentation in man lxc.conf:
If the rootfs is an image file or a device block and the fstab is used to mount a point somewhere in this rootfs, the path of the rootfs mount point should be prefixed with the /usr/lib/lxc/rootfs default path or the value of lxc.rootfs.mount if specified.
Anyway this should be fixed in 0.8.0_rc2-r2 which is now in tree, I’ve not been able to test it thoroughly yet, so holler at me if something doesn’t work. [Less]
|
|
Posted
about 1 year
ago
by
Mu Qiao (qiaomuf)
Glad to see that libbash is accepted by Google Summer of Code this year. Congratulations to André Aparício! Petteri and I will both mentor this project during the summer. Before GSoC begins, I’d like to explain the current status of this
... [More]
project.
I worked on this project in GSoC 2011. After the summer ended, I spent months seeking a job so I didn’t pay much attention to the project at that time. And later, I made several commits to the project and it’s looking better.
So far, we can generate correct metadata for 13849 ebuilds (5435 more than before). Nearly half of the ebuilds in the main tree can be handled by libbash now. So the goal of this year’s GSoC is to make libbash be able to handle the rest of the ebuilds.
What’s stopping us from handling more ebuilds? There are several eclasses that our grammar file can not parse. That means any ebuild that inherits from these eclasses can not be handled by libbash. On the other hand, the runtime part is not fully implemented, which is due to lack of grammar support or low priority. I have to say it’s hard to tell all the problems we are facing at once. I have been working in this pattern for quite some time:
1. Use our implementation of instruo to generate metadata for the ebuilds in the main tree.
2. Sort the error output to see which errors are the biggest enemies.
3. Solve the problems that are figured out from the error output
Generating ebuild metadata will take about 10 minutes so this process is not fast. I need to solve as many problems as I can at one pass. Typically new errors will occur after I fix the old ones. And the new errors are not introduced by the new commits. They were just hidden by the old ones. That’s why it’s hard to move forward and handle more ebuilds in a short time.
In future, the error output should be improved so it would be easy for us to figure out more problems. After constantly working on the error output for some time, I’m sure we will get fewer errors and handle more ebuilds. Looking forward to seeing a working libbash for all the ebuilds at the end of this summer.
[Less]
|
|
Posted
about 1 year
ago
by
Michał Górny (mgorny)
If you’re designing an XML-based data format, then I beg you, please read the few following rules and obey them. XML may look easy, and even is easy but that doesn’t mean that writing a good one is. And if you’re going to invent second
... [More]
HTML, then please, just use JSON or any other random container. That will be easier for you, and easier for us.
1. Thou shalt always write a schema
Every XML format should be well described. And no, your ten-stanza poem is not enough. Complete, dedicated Wiki neither. These usually describe nicely (or less nicely) how to write your XML. That could be great if that’s all you’re interested in. But if that’s supposed to be some public format, there is one more important thing…
It’s called reading. Or parsing. Or just transforming. If you need to handle random XML files, coming from various sources, written by random people, you have to know what you can expect and what can you assume. It’s not enough to say what <x/> does — I need to know where it can appear and what I can find inside.
There are already well-deployed XML description formats such as DTD, Relax-NG or XML Schema. Please use one of them, I will be grateful. Not only they describe the format strictly and accurately but they also provide a very simple means to validate XML files. It’s helpful both to us, who parse it, and to people who actually write such XML.
An XML without spec is an XML where every element can appear anywhere in the document. In other words, it’s not even XML but an ugly tag soup.
2. Thy XML shalt be structured, not flat
XML provides means to create neat, hierarchical structures. Use them. If your documents consists of logical parts like sections or chapters, put their complete content in a single <section/> or <chapter>, or any other thing that may come into your head. That’s the correct way of doing that in XML.
Random headings and separators are not enough. Even if your spec says they always and definitely start a new section, that’s not enough. If you don’t believe us, try splitting that thing into parts yourself. Especially when you have sub-headings, sub-sub-headings and so on.
A flat-structured XML is no real XML. It’s just a text file with a few unnecessary elements.
3. Thou shalt split text into blocks using XML, not text delimeters
Even if you think that’ll make writing much easier, do not ever try to use simple character delimiters to split text into blocks. If you need a list, create a list of XML elements. Like the following:
<l>elem1</l>
<l>elem2</l>
<l>elem3</l>
And yes, I know elem1,elem2,elem3 is shorter and easier to type. But guess what — it’s hell to parse. It isn’t even XML — you either have to handle it externally or create a complex recursive template which will split it and handle each token separately. That’s very bad.
An XML which uses random delimeters to create lists is no XML. It’s called CSV.
4. Thou shalt not allow insane structures
Even if you think noone will create an insane structure in your document, it’s not enough. Saying it’s disallowed on your awesome Wiki is not enough either. Forbid it if it’s supposed to be forbidden.
Otherwise, someone finally will use it. He or she will deliberately ignore your warning because it works. And even if they don’t, we will have to support it anyway in a compliant parser.
If you expect your data to be interchangeable with widely used formats, take a look at them. Don’t allow insane things which none of these formats do — or we’ll have to either refuse to convert some files, convert them incorrectly or waste our time writing complex blocks converting them to sane ones.
Simply, don’t do it. Even HTML doesn’t do that… well, that much.
5. Thou shalt write readable XML, not bytecode
The major point of using XML is that the data is both readable to machines and humans. Leave it that way. You have the whole human language at your disposal, so don’t write zeros, ones and other random numbers which are explained on your great Wiki.
Say, an attribute called type should actually name some type. Say, article can be some type. 1 usually ain’t. And if that type only describes width of indent, then name it so! Calling it a type is as useful as calling it a thing. Or some-other-thing and a-third-thing.
XML without human-readable text is no XML. Hell, even byte-compiled XML should have readable element names! That’s the whole point with it. Otherwise, you just end up developing another custom, useless format. [Less]
|
|
Posted
about 1 year
ago
by
Steve Dibb (beandog)
It’s been a long time since I’ve worked on much anything computer-related as a hobby. Things have changed quite a lot in the past year. I moved to a much smaller apartment in Salt Lake, which is about a third the size of my old place. The
... [More]
idea was to get trim the fat and focus on going back to school, which is my major direction in life these days. When I moved in, I didn’t have room for setting up a desktop computer anywhere, so it’s been just my netbook and me. That suits me plenty fine, though, I wasn’t really using it that much either. I had just upgraded to a six-core so I could rip DVDs much faster, and now it was sitting headless wherever I could find room, and even then, only used occasionally.
It’s not just at home that things have been changing. At work I got to make the transition from programmer to full-time sysadmin, and I’m absolutely loving it. I knew I was getting tired of coding, and I had always enjoyed just taking care of servers, and now I get to do that all day long. When I initially started as a sysadmin, I didn’t think our small company would have enough work for me to do after a few months. In actuality, I’m kept busy all the time. The part I like the most is that part of my job is doing research, how to do things better, more efficiently, anything to make the workload easier. It’s fun.
On top of all that, my school attendance is starting to ramp up more, and I’ve been consistently drifting to adding more classes to my workload. All this stuff has basically booted Linux out of my life as a hobby, and so now I need things to “just work” without hassle, so I leave my installations alone.
One thing I’d been neglecting a little bit was my entire HTPC setup. I hadn’t been using it much lately just because I would mostly stream some Netflix (yay, Doctor Who!). My setup has been a beast though, normally running for months on end without the slightest hiccup. What started to happen though is that I would come back to using it, switching my HDMI input over, and the box would be powered off for some reason. Most of the time, I would either power it back on and go on with life or just ignore it. Until one day it wouldn’t power on at all, and I just shrugged it off and determined to look at it later.
Well, later turned out to be finals week, when my brain has been working overtime, and I seriously needed a hobby. I pulled out my main frontend and started looking at it to see what was going on. It was plugged in properly and everything looked legit, but when I hit the power, the CPU fan would start up for a second and then everything would stop. After fiddling with it for a bit, I started to notice that something was smelling burnt. Once that happened, I abandoned my diagnosis. Even if I did manage to get it working, I didn’t want it to catch everything on fire.
At the same time, my external USB drive enclosure died on me. So even if I could have gotten it working, I still wouldn’t have had a way to watch my shows. Them giving out on me hasn’t bothered me in the least — the entire setup has been running flawlessly for years, and I’d managed to get a lot of mileage out of them.
Now I had to decide what I was going to do. I have a lot of hardware, but in pieces. I have four mini-ITX boards altogether, two of them are VIA C7 chipsets, and the other two are Zotac boards both running low-powered Celeron CPUs (around 35W if I remember correctly). The power supplies for the VIA boards use 20-pin connectors and only run at about 80W, and aren’t enough to handle the Zotac boards which use 24-pin connectors. So I have this mix of hardware, and nothing powerful enough to act as a frontend.
There are some great packaged systems out there now where for between $200 to $300 you can get an entire package in one go that does exactly what I’m putting together myself. I considered the idea of just starting over, but I decided that it’d be cheaper to just salvage what I could.
So this week I ordered a new USB HDD enclosure, and I also ordered a new power supply for the main Zotac board. I found a site that sells really small power supplies for mini-ITX boards, called picoPSU. The design eliminates a lot of the hardware that I would normally need to get all the power to my box. I was really skeptical about them when I first heard of it, but did some looking around and it looks like it’s exactly what I need.
In the meantime, I ripped out my motherboard out of my desktop, and put both Zotac boards in there to make sure they still work, and thankfully they do. I got the old setup pieced together using my desktop case, and fired up the old system to play around with it.
I had started to forget how much time I put into this thing. I forgot that I had put countless hours stitching this thing together, running a custom build of Gentoo suited to run on small environments. On top of that I made hacks to mythvideo and got those working to polish off some rough edges. It just started to come back to me how much I’d worked on this … and how much fun it was.
I played around with my frontend a little bit, and fired up a few movies just to try out the surround sound. It was awesome. I’d forgotten how nice it was to have that huge library on demand, too.
So I’m excited now to get things up and running. It’s been a good little while.
[Less]
|
Copyright
©
2013
Black Duck Software, Inc.
and its contributors, Some Rights Reserved. Unless otherwise marked, this work is licensed under a
Creative Commons Attribution 3.0 Unported License
. Ohloh
®
and the Ohloh logo are trademarks of
Black Duck Software, Inc.
in the United States and/or other jurisdictions. All other trademarks are the property of their respective holders.