Pythonic Wisdom

Best flash file system

2011-11-28T00:49:00.000-08:00

A while ago I wanted to get most theoretical flash lifespan out of the following stack

app / sqlite / file system / linux / consumer flash

Background here, to summarize, writes smaller than erase page size put just as much strain on the flash as exactly one erase page. Typical erase page size for 1GB~4GB consumer flash is 64KB or 128KB.

Modern flashes implement some form of wear leveling, yet no consumer flash manufacturer specifies what algorithm they use or whether it works over entire flash or several logical blocks of flash. Some specs are available for industrial flashes, but those are pricey and, it seems, fall behind times, that is consumer flash cards are built on newer tech and ought to yield better results.

The app was modified to (read -- compare -- write if needed) in transaction, and to request possible changes in batches, thus removing unnecessary writes and combining multiple small changes into fewer larger writes.

SQLite is a practical necessity, I want transaction safety as a guarantee of logical data consistency. SQLite has options though, these were tested. A 100 byte row takes 1KB in the database, the difference is comprised of indices, data structures and metadata.

Linux offers many file systems, some more developed than others, yet most were made for hard disk storage. As my type of flash is seen as a block block device to the system, I cannot use dedicated flash file systems anyway.

Result key:

test1: update 1000 records 100 at a time
test2: save 1000 records 1 at a time, delete 1000 records 1 at a time
--
total: total KB written
bins: 128KB-size erase blocks touched
worst: number of time most used erase sector overwritten
bins >90% worst: number of erase blocks overwritten >90% of worst

Raw results:

sqlite std options                against ext3 std options
total     1288K, bins       15, worst       11, bins >90% worst 3 # whitelist
total 106836K, bins     266, worst   2001, bins >90% worst 2 # offline
sqlite page size 64K           against ext3 std options
total     7740K, bins       26, worst       14, bins >90% worst 2
total 616132K, bins      267, worst   2046, bins >90% worst 4
sqlite journal_mode persist against ext3 std options
total 992K, bins 11, worst 21, bins >90% worst 1
total 48016K, bins 118, worst 4004, bins >90% worst 1
sqlite synchronous off         against ext3 std options    (unsafe)
total      292K, bins       12, worst        7, bins >90% worst 1
total    3748K, bins       25, worst       88, bins >90% worst 1
sqlite write ahead logging    against ext3 std options
total     7740K, bins       26, worst       11, bins >90% worst 4
total 616100K, bins      267, worst   2046, bins >90% worst 4
--
sqlite std ext3 std (repeated for comparison)
total 1288K, bins 15, worst 11, bins >90% worst 3 # whitelist
total 106836K, bins 266, worst 2001, bins >90% worst 2 # offline
sqlite std ext3 std noatime
total 1312K, bins 16, worst 11, bins >90% worst 4
total 106616K, bins 266, worst 2002, bins >90% worst 2
--
--
sqlite best (std; persist)
total     1288K, bins       15, worst       11, bins >90% worst 3 # std
total 106836K, bins     266, worst   2001, bins >90% worst 2
total      992K, bins       11, worst       21, bins >90% worst 1 # persist
total 48016K, bins      118, worst   4004, bins >90% worst 1
--
sqlite std                         against ext4 std options
total     1280K, bins       18, worst       10, bins >90% worst 2
total 106768K, bins      296, worst   2000, bins >90% worst 1
sqlite std,pers               against ext4 ^journal
total      904K, bins       14, worst       13, bins >90% worst 1 # std
total   42996K, bins      23, worst    2015, bins >90% worst 4
total      820K, bins       11, worst       20, bins >90% worst 1 # persist
total 34508K, bins        7, worst    4000, bins >90% worst 1
sqlite std,pers,page    against ext4 stride 128K
total     1284K, bins       19, worst       10, bins >90% worst 3 # std
total 106852K, bins      266, worst   2000, bins >90% worst 1
total    1012K, bins       12, worst       20, bins >90% worst 1 # persist
total   58296K, bins      197, worst   4000, bins >90% worst 1
total     7648K, bins       29, worst       11, bins >90% worst 2 # page size 64K
total 600220K, bins      267, worst   2022, bins >90% worst 3
--
sqlite std                         against nilfs2
total 319800K, bins    2500, worst       20, bins >90% worst 1
total 255528K, bins    1998, worst       18, bins >90% worst 1 # nilfs2 ran out of disk space - it's gc bugs
sqlite std, persist         against nilfs2 protection period 1 second, other tweaks
total     2308K, bins       21, worst        3, bins >90% worst 1 #std
total 319580K, bins    2498, worst      20, bins >90% worst 1
total     2804K, bins       24, worst        3, bins >90% worst 1
total 166032K, bins    1300, worst      10, bins >90% worst 1 # incomplete nilfs2 crashed
total     2592K, bins       23, worst        3, bins >90% worst 9
total 707968K, bins    5532, worst      27, bins >90% worst 1
sqlite std against btrfs std
# too raw - ran out of space copying in static image and code
sqlite std                         against fuse-exfat
total      116K, bins        9, worst        1, bins >90% worst 9
# fuse evidently doesn't sync
total        48K, bins        5, worst        1, bins >90% worst 5

/etc/init.d/jenkins

2011-09-07T04:15:00.000-07:00


#!/sbin/runscript

depend() {
        use net dns logger
}

checkconfig() {
        return 0
}

start() {
        checkconfig || return $?
        ebegin "Starting jenkins"
        start-stop-daemon --start --background --user dev --env JENKINS_HOME=/var/lib/jenkins \
                --make-pidfile --pidfile /var/run/jenkins.pid \
                --startas /usr/bin/java -- -jar /usr/share/jenkins.war --httpPort=8014 --prefix=/jenkins
        eend $? "Failed to start jenkins"
}

stop() {
        ebegin "Stopping jenkins"
        start-stop-daemon --stop --background --user dev --env JENKINS_HOME=/var/lib/jenkins \
                --make-pidfile --pidfile /var/run/jenkins.pid \
                --startas /usr/bin/java -- -jar /usr/share/jenkins.war --httpPort=8014 --prefix=/jenkins
        eend $? "Failed to stop jenkins"
}

What is Google SRE, a.k.a. google.com engineer?

2011-06-28T17:14:00.000-07:00

A few months ago I went to PyCon, Google had a booth and a riddle there. I was sort of looking at Google as a potential employer in Europe for a while prior to that, so I had a chat with two Google recruiters. Unfortunately neither of them could tell me much about their European offices, nevertheless it took off from there.

Unfortunately the more proactive recruiter was after new meat for SRE, he emailed:

"My name is Xxx and I'm a recruiter at Google. I support the Google.com Engineering Team."

I wondered what different positions at Google were for a while, so I already knew that "google.com engineer" equals "SRE" equals "site reliability engineer" equals lots of hacking and very little engineering. I have read about reliability at google before, including a very insightful report on hard drive reliability.

From then on I had a few phone contacts, handed off to European recruiter, 3 phone interviews, and eventually an invitation for an on-site interview day at GooglePlex in silicon valley. That was exciting!

Unfortunately the SRE tag stuck with me from the start and I was unable to shake it off. Everyone who interviewed me on the phone was an SRE, everyone who interviewed me on sitewas an SRE or former SRE.

Out of on-site interviews 1 was magnificent and all others sucked.

The great interview was conducted by a slightly older googler who had great command of algorithms and asked me a question that somehow my university never did. O(n) solution to a given problem was already discovered in the 70s, but somehow it was not in my curriculum even though many other similar problems were. Needless to say it was challenging. Also I think I didn't do so great, although I doubt I did awfully badly either.

The other interviewers were around my age and asked lame and redundant questions like "what's in an inode?" Lame because it's mostly trivia, redundant because I was asked that on the phone before. One SRE interviewer (on the phone), when asked to describe what working SRE was like, eventually gave up and told me "it's a job." I mean, really, if you are considering work for Google almighty, would ever take an offer of "a job?"

Coming back to the subject of this post, what is SRE position and what do SREs do? Basically SREs have to keep the crap that real engineers wrote running. They don't really code, they don't really do much engineering. Yet, they are more than server-rebooting-monkeys. And of course they have to make the tools to make their job easier. Plus they are sometimes on call, and if something breaks they have to repair it pretty fast. Yes they have to be on top of things, yes they have to be hackers at heart, and yes the job could be interesting.

Still I couldn't help but wonder -- do I want to create or merely help run other people's stuff?

Moreover I couldn't shake off the feeling that recruiters tried to suck me into the quagmire of SRE regardless of my real skills.

So here comes advise:

If you are applying for Google and hear SRE, run.

Redo your resume and remove any references to hardware or server maintenance, that's a red flag that puts you in SRE category immediately.

Most importantly, think hard what you want to do.

Coming up, Brazil-esque vision of real google engineers and other things that sucked.

I would like to finish off on a positive note, I really really liked the full-height "barn" automatic bike doors buildings in Google campus are equipped with. It just warmed my heart to see that someone thought of that!

PyCon US 2011 review

2011-03-19T19:07:00.000-07:00

Overall

To be honest, I still have not made up my mind about the conference. There was a lot, some was great, some was not. And I drove to and from, 4 days each way, which was an experience on its own.

Atlanta turned out, quite unexpectedly, a very nice city, remarkably different from the rest of US of A, for one the streets are not laid out in a grid (although of course there are some) and with the help of my couch surfing host, we went to some really nice, authentic and interesting places to eat. Most important to note, a Korean-flavoured taco place, the runner up is an authentic Cuban joint, that serves sandwiches, burgers and most importantly sucos, kind of like smoothies. Had I stayed in the hotel, as the conference recommended, I would have missed so much! Kudos to William of couch surfing in Atlanta!

Tutorials

I picked only one tutorial, as I wanted to find out if tutorials were worth it. As such I picked the most appealing to me, on machine learning. First half was really good, perhaps just a tad too slow for me, as it allowed to go and refresh my classification concepts on Wikipedia, but when I tuned back in, I have already missed a little bit of the talk. Second half was a bit confused. The code examples didn't seem to complete in reasonable amount of time and it seems almost none tried to run them, talk was more on how it's done with this particular library and less on why it's done or why Python is involved at all. These were nice 3 hours, although to think that I shelled out 100 bucks for the tutorial, I expected more.

Keynotes

I missed the PSF chairman address, but was on time to see the 1st keynote, delivered by Hilary Mason. That was a downer. Sure, there was a kitten and a dubious, but fun, statistics on pycon web site. But there was no content, absolutely zip! It is as if the speaker was instructed to dumb down the address so that a complete stranger who walked in to the wrong hall would understand. After all she's not a bad speaker. And to think that I could have had breakfast instead!

"Chat with Guido" was more interesting. I wouldn't say it was super, as apparently Guido is not as concise in his speech as in his programs, but it was interesting nevertheless.

Sunday didn't seem to feature a speech deserving to be called Keynote. Threadless was interesting to listen to, the rest was entirely forgettable.

Talks

Now that all is attended and done, very few talks come to mind. In all honesty, very few were memorable enough.

Friday started with a rather interesting and forcefully delivered talk "Getting the job," that focused on non-technical aspects of getting hired, in particular to a company that uses Python, loves open source and hiring is done by a regular manager. It emphasized social skills and left me with impression that a lot needs to improve in software companies.

The other interesting talk was, remarkably "Javascript for Pythonistas." That should say something about a Python conference if one the more interesting talks focuses on a "competitor" language.

I must have missed a couple of talks, as I went to talk to all the interesting companies at the startup row.

I was really looking forward to Zed Shaw's talk on ZeroMQ on Saturday, based on his spectacular, stunning talk "ACL is dead", that includes part on "how to keep your soul" (in the industry). This time it was not as great, perhaps because the allotted time was significantly shorter.

Unfortunately I missed Augie Fackler's talk on choice of http libraries in Python. I have seen him prepare earlier and must have been a great talk, but I figured I'd check other talks first, becuase surely the beginning of his talk would be too basic. Well by the time I got there, all I heard was "that was it, do you have any questions?" He only used half his slot, and there were no questions. Or none that I remember. I can't wait for all the videos to appear online so that I can check it out.

Sunday was a total bummer. There were lightning talks in the morning, at least one of which was good, obligatory business-like talks and 3 time slots for normal tracked talks. First of these features a rather non-issue talk masterfully delivered by Raymond Hettinger, and the next 2 slots there was obsolutely nothing worth staying at the conference for. The half-day was concluded with more lightning talks, at least 2 of which stood out, I'll post links when videos show up, and introduction to sprints that was downer too. I should have realized this in advance and gone to see the city.

And the ones that I mentioned were the good talks. More than once did I find myself bouncing between one room and next and back, just because there was nothing worth listening to!

Goodies

PyCon featured a swag shop, ran by Elegant Stitches, there were a couple of cool t-shirts, "flash wound" with a keyboard jammed in a broken monitor and a neat Python with an apple tee, perhaps referring to the tree of knowledge. Some swag was good, none was great. As with much tech swag, there was conspicuous lack of colours, women's cuts or even some sizes.

There was also a book store, with hoards of books on Python and related subjects, what surprised me the most was that all books except for one (and I checked) had code examples in serif font, most as far as I remember didn't even bother to highlight the keywords. Some went as backwards as include screenshots of IDE. I could perhaps accept that in an old Fortran book published in the age of punch cards, but nowadays surely programming is done on a display with syntax highlighting and therefore programmer is accustomed to seeing her code in good sans-serif font, in colour, keywords in bold and perhaps comments marked in some way (though I'm strictly against italics and underlined text). Surely then modern programming books must adopt same convention! If printing in black and white saves money, sure, but the rest?

In Conclusion

Will I go again? Only if someone pays me to.

Should you go? Well that depends:

If you practiced Python for years, don't go, you won't learn anything.
If you are a student, it's too expensive, though if you get funding, go for it.
If you have many friends to meet at the conference, sure, go ahead.

Going to PyCon

2011-02-02T15:21:00.000-08:00

PyCon 2011 will be held in Atlanta, GA, USA during March 9th~17th.

It's quite pricey though, at least if you pay yourself. Conference is 300 for early brids and tutorials are 100 each for same. Moreover the special hotel rate is 160 a night, what a ripoff!

It's probably a nice hotel and all, but if you read Trip Advisor, some manage to stay in same hotel for 53 plus tax a night! I didn't book a hotel yet, but there's a bunch of cheaper options in the vicinity.

PyCon expects over a thousand attendees, so perhaps it is worthwhile to compare to DrupalCon costs.

Anyhow, I'm going to make a Google calendar for the event because the PyCon's website leaves far to be desired, and I will surely post my impressions and more when I'm there!

Google, cough up!

2010-12-20T15:29:00.000-08:00

Google spends several b on data centers every year (short scale), it neither reports how much exactly nor on what, nor what share of the cpu load is eaten by Python programs, yet we know big G definitely loves and uses Python.

If Python cpu consumption totaled a meager 10% of total, and some project improved Python's performance by a meager 1%, that alone would save Google several m a year.

Meanwhile PyPy, by far the most fertile modern python performance project, received eurostars grant E! 4791 "PyJIT" worth half an m that is supposed to last several years. ( http://www.eurostars-eureka.eu/search.do )

Thus, Google, cough up!

Truly international Python

2010-11-30T09:00:00.000-08:00

Let's recall Guido's old Computer Programming for Everybody (CP4E) proposal.

Nowadays that Python is established, it's high time to push Python into education, especially first programming language education. I think, in the modern world it means pre-school.

Now the larger part of the world's children doesn't learn English before school, therefore we need to have truly localized Python.

Some might recall a Python derivative demo with unicode variable names (link anyone?).

I think we ought to go further. For example, consider imaginary language pig latin:


"""This does that"""  --> """Thiso acto thato"""   # docstrings
__version__ = (1,2,3) --> __versio__ = (1,2,3)     # variable names
import time           --> importo chrono           # standard module names
def foo(): pass       --> defo foo(): passo        # Python keywords
"foo".upper()         --> "foo".uppero()           # standard library
raise Xx("undefined") --> raisio Xx("indifinito")  # errors
#!/usr/bin/python     --> #!/usr/bin/pythono       # executable name
#!/usr/bin/python     --> #!/usero/binaro/pythono  # name and path

Of course there are concerns for many languages:

Each language needs to establish stable translations for keywords, basic types, standard modules, methods in standard modules, etc.
Some languages don't support word spaces natively
Some languages have different punctuation rules, e.g. comma for decimal point
Some languages use different quotes
RTL languages spell words RTL yet (some/all?) spell numbers LTR
Hopefully none has to recreate 10,000-separator system ;-)

Anyhow, it's not the issue of core Python to support particular languages, what is needed is:

the concept that this is needed, and
the base where from a particular localization can evolve from

Here, a fun example, how Python might look like in google-translate-simplified-chinese. Blame google, not me as I know very little about this language.


"""This does that"""  --> """这是"""           # docstrings
__version__ = (1,2,3) --> __版本__ = (1,2,3)   # variable names
import time           --> 进口 时间             # standard module names
def foo(): pass       --> 业 美孚（）： 通过     # Python keywords
"foo".upper()         --> “富” 上层（）         # standard library
raise Xx("undefined") --> 提高。二十（“未定义”） # errors
#!/usr/bin/python     --> #!/usr/bin/蛇        # executable name
#!/usr/bin/python     --> #!/用户/二进制/蛇      # name and path

I track this here and will update with the received feedback:

http://pythonic-wisdom.blogspot.com/2010/11/truly-international-python.html

Literal name lookups

2010-11-29T18:09:00.000-08:00

Todo and caveats

dictnotes.txt talks about sparse dictionaries for a different reason, check if spareseness is in fact implemented in the newest code.

tradeoff between memory (sparse dicts) and time (dense dicts) is not always obvious. perhaps some dicts could be targeted for extra sparseness live?

locals() is not really a dict since python 1.something.

python string object caches own hash value already, this could be improved though, rarely executed code could recompute hashes faster than it takes to load extra 4 or 8 bytes from memory and reduce cache pressure, there's probably little reason to store precomputed hash in pyc though.

This post will track the status of my attempts to understand and hopefully optimize name lookups in python.

Consider the following contrived example


aglobal = ""

def outer():
def inner():
alocal = 3
li = [str(acomprehension) for acomprehension in range(aclosure+alocal)]
return aglobal.join(li)
aclosure = 5
return inner()

Now, consider what name lookup happens inside invocation of inner():

alocal and acomprehension: locals() -> hit
aclosure: locals() -> miss, closure -> hit
aglobal: locals()-> miss, closure -> miss, globals() -> hit
str: locals() -> miss, closure -> miss, globals() -> miss, builtins -> hit

Now these namespaces are implemented internally as hashmaps, that might look like this:


# builtins hash table (using python syntax rather than C)
idx = hash(name) % len(hashtable)
hashtable = [
[("int", {pointer to int})],              #
[],                                       # some entries will be empty
[("float", {pointer to float})],          #
[("str", {pointer to str})],              # what we are looking for
[("unicode", {ptr}), ("True", {ptr})]     # example collision
]
items = hashtable(idx)
for item in items:
if item[0] == name: return item[1]        # actual string comparison

raise NameError("name '%s' is not defined" % name)

The larger the nesting (locals, number of closures, globals, builtins), the more important it become not to find a name in a hashtable, but to decide quickly that the name is not in the table.

Therefore the intermediatory talbes be sparse like this:


hashtable = [
[("int", {pointer to int})],              #
[],                                       # many entries will be empty
[],                                       #
[],                                       #
[],                                       #
[("float", {pointer to float})],          #
[],                                       #
[("str", {pointer to str})],              # what we are looking for
[],                                       #
[("unicode", {ptr})],                     #
[],                                       #
[],                                       #
[("True", {ptr})]                         # no or few collision
]

Moreover hot hashtables (those that are looked up in frequently) should be instrumented with counters for hits, early misses, and late misses (requiring full string comparison).

Wicked graphics!

2010-06-07T03:46:00.000-07:00

Circle inner radius 50, outer radius 51
Circle inner radius 50, outer radius 51.5

Wicked!

Cython, the new frontier

2010-06-02T01:05:00.000-07:00

Learning to write in Cython, a novel alternative to writing native python extensions in raw C.

Stand by for updates...

Meanwhile an overvire of the many ways to interface python with native programs and libraries:

integrate through files, sockets, pipes or even mmap and shm
write extension in C
write extension in C and wrap it in Python module
write module in mix of C and Python and translate with Cython
use native library via ctypes
use native library via ctypes and wrap in a Python module
generate wrappers with SWIG
write all in Python, numpy for performance

1^st is the easiest to start with, but doesn't offer tight control over native program's execution, such as timing, errors, feedback.

2^nd and 3^rd are quite sane, and after a bit of learning even quite easy to debug. The advantage of a Python wrapper is to split the 2 parts of integration, the C and Python so that programming is the easiest.

PICs suck

2010-01-28T13:50:00.000-08:00

I've been battling a little PIC for a couple weeks in my spare time...

Why did I chose PIC over AVR? Now the question buffles me again. Truth be told, AVRs don't seem to frequent multiple serial ports on a small chip. The price of simplest PIC with 2 UARTS was twice less that of simplest AVR with 2 UARTS. That was basically why.

However for the 2 USD I won having made the choice I seem to have to pay back in spades.

Why could I think of AVR? The most important is the compiler. If the wise GNU people already spent almost 23 years writing it, why not use it? Indeed AVR is the right way. I thought a proper recommended compiler for PICs, the C18 could not be so bad. Not so.

A quick example. I want to play with 2 serial ports, in C18 I have to use separate functions DataRdy1USART and DataRdy2USART to see if I'm getting something on the serial lines. Worse then, if I only had 1 serial port on a pic I'd have to use a third function, DataRdyUSART. Let alone the slightly abbreviated, two-thirds-camel-case-one-third-all-caps names, if I were to check the error state, I have to read from the uart, and then the crown jewel -- say I wanted to output a string of text, C18 library provides a function for that -- puts1USART (notice yet another capitalization scheme), which outputs a null-terminated string... including the null character!

Then let's consider a function DisablePullups. A programmer would think a bit and start digging what its arguments might mean - which port? what pins? Alas! it only works on PORTB and on all pins at once.

Someone said PICs have best documentation. Not quite so. I've encountered several mistakes and in the first PIC datasheet I ever worked with and it seems that trying to contact the documentation team is not as easy as it was made out to be. Through the powers of google, find in files, find in pdf, google again and the hordes of others who have trod this path before me I was able to resolve all the issues I had so far, but it took much precious time.

I'm afraid these gotchas that got me are just the beginning, meanwhile the battle rages on...

Next time I start with AVR and see where that leads me!

P.S. PICs have many advatanges, easier access to multiple serial ports is just one of these. If you wanted CAN or Ethernet on cheap, AVR got just 1 practical model, PIC got many, and I bet these are just the beginning too.

USB HID with Linux and Python

2009-07-07T01:33:00.000-07:00

I posted some working python code in python recipes section on activestate, here's link:

http://code.activestate.com/recipes/576834/

I figured i post it now, because I haven't done any device-specific changes yet, and I'm about to.

There is an enormous void of hiddev programming information out there... "in hiddev space none can hear you scream" kind of thing. There's some for usb hid event devices, but none for hiddevN devices.

If anyone wonders, the usb hid device I'm working on is made and sold by Delcom Products (I have a strange feeling it used to be Delcome Engineering before). It looks like a good device, and when I get to the bottom of it I'll post more.

Battle of Mind with Dell DRAC

2009-05-26T01:40:00.000-07:00

Dell's DRAC allows you to control your server through an ssh console or a web browser.

At least in practice.

In reality it goes more like this:
1. Start with ssh, log in, look how cool it is, get confused...
2. Try to find something on the internet...
3. It's not quite there, but try again, go to step 2.
4. Figure what the hell, it can't be so hard, but alas! you session timed out, to log in go to step 1.
5. Consider thy art a hacker this lowly dell must kneel before thee, but alas! session timed out again, process to step 1.
6. After a night's rest you find a few PDF's on Dell site (by great wizardry of Google, of course), and decide to check the browser access instead.
7. Log in.
8. Navigate to Media section and get thoroughly confused unable to make a distinction between Virtual Media and Virtual Flash. Besides it keeps bugging you to install something.
9. Find another PDF on Dell's site (-"-) and get a clue: it's almost like vmware, except the server is real. somewhere in the dungeon of ISP.
10. Install the extension for Virtual Media and alas! Firefox needs a restart.
11. Restart and find out that your session is dead - go to step 7.
12. Select your ISO image, and wait... what's wrong with that button, why is it still disabled?
13. Use your grand hacker mad skillz and figure out that clicking around randomly might help.
14. Try 13 again.
15. And again.
16. And again.
17. Find out that the whole Virtual Media blob is disabled in Configuration.
18. Re-enable, re-select file, a nice pop-up tells you now to tweak javasctipt (?) settings, nice! But what now?
19. Proceed to Console, it wants you to install another extension...
20. Install extension.
21. Restart Firefox.
22. Session is dead, go back to 7 again.
23. Go to Console, all looks good, but where is the console?
24. Click around randomly.
25. Bah! Firefox died (2nd time already actually, but no matter)
26. Start Firefox again, session is dead, back to step 7.
27. Then by the virtue of true madness you see (if you are lucky that is) in the console that started Firefox errors like "blahblahblah not found"
28. Run some find .mozilla -name blah, and discover that....
29. The script that starts a standalone viewer program assumes that your profile is called "default."
30. Restart Firefox in default profile, session dead, go to step 7.
31. Install extension in default profile, restart Firefox, session dead go to step 7.
32. A nice popup tells you how to tweak here-be-dragons settings in Firefox, nice!
33. A console viewer shows up, telling you that operating system is not installed.
34. Go to power, reset, really sure to reset, apply...
35. Watch the console screen blink green for a while... and a while longer... and still a while... and whoa BIOS boots!
36. See press F2 for BIOS config, nice screen, perhaps go to boot options?
37. Oh horror! Arrow keys don't work!
38. Muck around long.
39. Google around even longer.
40. No luck, try keypad arrow keys - salvation!
41. Meanwhile browser session timed out, but now there's an overlay screen prompting you for a password, nice!
42. Decide to check your RAID settings, arrow keys don't work and here, they don't support keyboard arrow keys anyway!
43. Think
44. Consider perhaps RAID settings are fine anyway...
45. Press Escape to quit, get a pop-up [ok] [cancel] with cancel selected. To move to [ok] you need arrow keys...

To Be Continued...

Booting linux on eBox-3300 a.k.a. vortex86dx

2009-01-30T07:26:00.000-08:00

Say you wanted to migrate from vortex86sx to vortex86dx. And there's a good reason to do so, you get floating point coprocessor and triple your cpu frequency. So you get yourself an eBox-3300 to replace your old eBox-2300SX. However it was not as easy as that.

First, compactflash is now secondary ide controller, thus /dev/hdc1 instead of /dev/hda1. Furthermore even through the peripherals appear exactly the same, you suddenly get DMA timeouts. Apparently it812x driver for the embedded IT8212 controller is rather buggy and wants to select some DMA mode even if BIOS recommends PIO and even with the kernel patch from DM&P that worked just fine on vortex86sx.

Thus here's a solution for all the hungry embedded linuxers out there.

Go to BIOS, find IDE settings and set DMA to lowest (and slowest) possible value. It sounds like enabling DMA when you mean to disable it, however linux is smarter than you are and it will figure out that lowest level of DMA is worse than auto (and thus highest) level of PIO and happily choose PIO for you.

That's all!

Links:
Vortex86SX and vortex86dx can be found here: DM&P (Taiwan)
EBox-3300 and eBox-2300SX can be found here: CompactPC (related to DM&P)

Love Grease Mokey, Adopt a Fire Bug ;-)

2009-01-08T06:53:00.000-08:00

Just now it took me about a minute to fix an annoying bit on RS Electronics website, namely the "family" selection box. Someone at RS had a great idea to limit the height of this box, which worked nicely as long as you are browsing a small product line, but when you get to the holy grail of component selection, connectors, you are presented with a list of 43 options, of which only the top two lines fit in the box height, thus you gotta scroll, and scroll a lot, because the these options are not grouped by pitch, so same pitch connectors happily live in several lines of the list, thus if you wanted to see all the options for 5.0 and 5.08 mm pitch you'd have to scroll up and down and up and down to get 3 lines of options through 2 lines of screen.

Firebug: F12 to turn on, right click in the box to inspect element, check CSS of element and its parents for abnormal height restrictions.
Google: greasemonkey add css, copy.
Greasemonkey: add new script, asterixize the url, paste, enter class and property name, inherit.
Reload.
Done.

Accessing Parallel Port on Linux from User Mode (2002)

2008-11-07T08:05:00.001-08:00

there are two things you can do from userland -

* access pc-style parallel port directly through inb/outb
. works only on standard pc parallel port
. most efficient (see below)
. can only be done by root

* access raw parallel port through ppdev (CONFIG_PPDEV)
. needs ppdev in kernel
. access control by normal file permissions (default root only)
. can be somewhat slower

* write a discipline driver, like lp
. needs kernel module or patch
. no real advantages over usermode


issues - speed:

the standard x86 parallel port hardware is still used through inb/outb,
whether you access it with inb/outb from usermode or you let kernel do it
for you (through ppdev or parallel port discipline).

inb and outb go though isa memory cycle (or are thes programmable
instructions?), I don't understand this fully, anyway, the fact is that no
matter how fast your cpu is, the time it takes to execute an inb or outb
is the same. Basically you, are limited to 300,000 outb's a second (not
sure about the exact number). No matter if it's 2.4 GHz P-4 or 100 MHz 486

ppdev special:
when you use ppdev you have to use two separate ioctl's - one to change
data direction (in case of bidirectional port) and all other control
lines/flags/etc, so if you need to change direction of data lines often,
this might mean your bandwidth is cut in half :(

irq special:
there's a provision for the parallel port hardware to send an irq when
status lines change (or some such), however you need to supply a module
parameter or kernel argument to enable this (on legacy x86 port at least),
because the logic that figures out the ioport apparantely doesn't dare to
enable the interrupt

thus if you want an interrupt it's same level of intrusiveness as
requiring a kernel module. yuck.

and without the interrupt you are destined to poll the parallel port
status. polling is a problem in it's own, because most kernel mechanisms
go through schedule, thus allowing only HZ precision (read 100 times a
second on standard x86).  If you want more frequent polls without
busywait, there's some info on the net about it.

issues - access:
inb/outb are only available to root, you could make a setuid program that
does ioperm() and then goes back to unpriviledged mode of course

ppdev is controlled by normal user/group and permission on
/dev/parports/0, etc, so it's quite usable

with your own parallel port discipline you can do anything you like

issue - portability:
inb/outb works only for x86 legacy hardware
ppdev and your a new parallel discipline should work on any paralle port
hardware.



Examples:

inb/outb - from whoever I\m replying to:
> #include 
> #include 
>
> int main(int argc, char **argv)
> {
>  int addr = 0x378;
>
>  if(ioperm(addr, 3, 1) == -1) {
>   perror("ioperm");
>   exit(1);
>  }
>  printf("0x%02x\n", inb(addr + 1));
> }
>
> The ioperm() only works if you're root.
2nd argument is 3 because there are 3 registers in standard parallel port,
namely data, control and status (don't remember in which order). If you
wanna use something more advanced, like ECP or EPP, you'll have more
registers.

ppdev:
#include 
#include 
void blah()
{
fd = open("/dev/parports/0", O_RDWR);
ioctl(fd, PPCLAIM, NULL);
arg = IEEE1284_MODE_BYTE;
ioctl(fd, PPSETMODE, &arg);

unsigned char c = 13;
int dir = 0;
ioctl(fd, PPDATADIR, &dir);
ioctl(fd, PPWDATA, &c);
}

you\ll have to read ppdev.c in the kernel sources to figure out the
ioctl's but it's not that bad :)



hope this helps,
dima

Korean Goodies

2008-06-06T01:24:00.000-07:00

Drama series
Korean drama series I liked enough to back up. One day I should write something about each.
bodigadeu
MISA
damo
all in
breathless
KSA
choonhyang + ost

Goodies Online
http://dic.impact.pe.kr/ A very good dictionary that users can contribute to. Used to be #1 or at least on the first page in google search for Korean-English dictionary. Not anymore. I can only wonder why -- granted commercial dictionaries became better, but still relegating this to the second page of search results is a crime! So here is my bit to keep it alive ;-)

Pythonic Wisdom Notebook

2008-03-12T00:40:00.000-07:00

Hérinne Thy ambihtere shalt keepe their Google notes. [link]